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Several  efficient  algorithms  for  image  recognition  end  segmentation 
and  a new  computer  architecture  for  image  processing  are  proposed.  The 
algorithms  are  "syntactic"  In  that  they  perform  structural  or  spatial 
analysis  rather  than  statistical  analysis,  and  a "grammar"  is  inferred 
for  describing  the  structures  of  patterns  In  an  Image.  Depending  on  the 
requirements  of  the  problem,  an  appropriate  grammatical  approach  is  used 
by  the  syntactic  algorithm. 

A finite-state  string  grammar  Is  applied  to  the  image  recognition  of 
highways,  rivers,  bridges,  and  commercial/industrlal  areas  from  LANDSAT 
images.  There  are  two  major  methods  in  the  string  grammar  approach  for 
Image  recognition;  namely,  the  syntax-directed  method  and  syntax-controlled 
method.  For  the  syntax-directed  method,  syntactic  analysis  is  performed 
by  a template  matching  which  Is  directed  by  the  syntactic  rules.  For 
the  syntax-controlled  method  an  automaton  .which  Is  directly  controlled 
by  the  syntactic  rules  is  used  for  the  syntactic  analysis. 

A tree  grammar  is  applied  to  the  image  segmentation  of  terrain  and 
tactical  targets  from  LANDSAT  and  infrared  Images  respectively.  The 
tree  grammar  approach  utilizes  a tree  automaton  to  extract  the  boundaries 
of  the  homogeneous  region  segments  of  the  image.  The  homogeneity  of  the 
region  segment  is  obtained  through  texture  feature  measurements  of  the 
Image. 


The  computer  architecture  proposed  Is  a special  purpose  system  In 
that  it  can  perform  an  Image  processing  task  on  several  picture-points  of 
an, image  at  the  same  time,  and  thus  takes  advantage  of  the  fact  that 
Image  processing  tasks  usually  exhibit  "parallelism".  This  architecture 
uses  a distributed  computing  approach.  Two  major  features  are  the  re- 
configurable  capability,  and  the  method  of  computer  exploitation  of  task 
parallelism.  Finally,  a parallel  parsing  scheme  for  tree  grammar  is  used 
to  demonstrate  the  higher  efficiency  of  the  proposed  computer  architecture 
than  the  conventional  parsing  scheme. 
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CHAPTER  I 

INTRODUCTION 

Image  segmentation  is  a computer  technique  that  breaks  an  image  into 
different  regions,  each  having  homogeneous  properties  £83].  In  image 
analysis  or  scene  analysis  the  desired  result  is  a computer-generated 
description  of  the  given  image  or  scene.  The  computer-generated  descrip- 
tion refers  to  specific  parts  (regions  or  objects)  In  the  image  or  scene. 
Therefore,  a first  step  Is  to  divide  the  image  into  these  parts;  that  is, 
to  do  image  segmentation.  Image  segmentation  Is  also  an  Important  stage 
in  sample  classification  [1],  and  image  compression  [83]. 

An  algorithm  is  a computational  procedure  showing  the  steps  the 
computer  Ss  asked  to  perform  [113].  Algorithms  Involving  statistical 
operations  are  statistical  algorithms.  Algorithms  which  perform  struc- 
tural or  spatial  analysis  in  image  processing  problems  are  called  "syntactic" 
algorithms  because  of  the  analogy  between  the  structure  of  patterns  and 
the  syntax  of  languages  [2], 

The  "language"  that  provides  the  structural  description  of  patterns 
In  terms  of  a set  of  pattern  primitives  and  their  composition  operations 
is  called  the  "patterh-descrlption  language."  The  rules  governing  the 
composition  of  primitives  into  patterns  are  usually  specified  by  the  so 
called  "grammar"  of  the  pattern  description  language.  After  each  primi- 
tive within  the  pattern  is  identified,  the  recognition  process  ts 
accomplished  by  performing  a syntactic  analysis  of  the  "sentence" 
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describing  the  given  pattern  to  determine  whether  or  not  ft  Is  syntacti- 
cally (or  grammatically)  correct  with  respect  to  the  specified  grammar. 
There  are  four  types  of  grammars  [2]  according  to  form;  namely,  type  0 
(unrestricted)  grammars,  type  1 (context-sensitive)  grammars,  type  2 
(context-free)  grammars,  and  type  3 (finite-state)  grammars.  “Finite 
state"  refers  to  the  fact  that  these  grammars  have  various  control 
states  (a  finite  number  of  them).  There  are  two  types  of  finite  state 
grammars:  string  grammars  and  tree  grammars  [13l.  If  a finite  state 
grammar  Is  used  In  the  syntax  analysis,  the  analysis  is  called  a finite 
state  grammar  approach.  The  selection  of  an  appropriate  grammar  for 
Image  processing  usually  depends  on  the  requirements  of  the  problem.  A 
syntactic  algorithm  for  Image  segmentation  and  two  for  image  recognition 
are  presented  In  Chapters  3 and  k\ 

An  algorithm  can  be  implemented  by  computer  software  (programs)  or 
hardware.  If  it  is  considered  thac  the  potential  usefulness  of  a specific 
algorithm  justifies  the  time  and  cost,  specific  hardware  can  be  built  to 
perform  the  algorithm.  That  is,  a special  small  computer  can  be  built 
to  perform  just  that  one  task.  With  the  ever  Increasing  interest  in,  and 
useful  applications  of  Image  processing,  a consideration  of  building  a 
special  purpose  computer  primarily  for  Image  processing  is  justified. 

Image  processing  done  on  large  computers  takes  a great  deal  of  memory 
space  and  is  very  time  consuming.  Thus,  the  cost  of  the  computer  time 
Is  high.  So  this  cost  has  to  be  weighed  against  the  cost  of  building  a 
special  purpose  computer. 

Image  processing  tasks  usually  exhibit  "parallelism."  That  is, 
hardware  can  be  built  to  perform  a task  on  several  picture-points  of  an 
image  at  the  same  time,  t.e.,  In  parallel.  This  makes  image  processing 
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as  the  method  of  segmenting  an  inage.  They  involve  region  merging, 
region  dividing,  or  a combination  of  merging  and  dividing. 

a.  Region  Merging 

This  has  been  done  by  five  approaches:  statistical,  linguistic, 
decision-theoretical,  relaxation,  and  interpretation  guided. 

(i)  The  statistical  approach  to  region  merging  has  been  developed 
by  Brice  and  Fennan  [3**];  Bajcsy  [393 ; Gupta  and  Wintz  [4o]; 
and  Jarvis  [114]; 

(ii)  the  linguistic  approach  has  been  investigated  by  Tsuji  and 
Fujiwara  [41]; 

(ill)  the  decision-theoretical  approach  was  developed  by  Yakimovsky 
and  Feldman  [4?.]; 

(iv)  the  relaxation  approach  was  developed  by  Rosenfeld,  Hummel, 
and  Zucker  [78];  and 

(v)  the  interpretation  guided  approach  has  been  investigated  by 
Tenenbaum  and  Barrow  [79]. 

b.  Region  Dividing 

This  involves  successively  partitioning  the  Image  by  certain  criteria 
and  was  first  put  forth  by  Robertson  [44],  and  Klinger  [45]. 

c.  Combination 

A combination  of  merging  and  dividing  has  been  proposed  by  Horowitz 
and  Pavlidis  [46,71]. 

3)  "Edge”  Detection  Methods:  These  methods  define  as  "edges”  the 
boundary  between  two  different  objects  in  a picture,  (for  example,  the 
"edge"  between  a human  neck  and  a sweater  neck)  and  consider  these  edges 
as  the  boundaries  of  the  segments  of  the  image.  There  are  several  approach^ 


as  follows: 
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a.  Template  Hatching 

This  approach  has  been  explored  by  Griffith  [47,48];  and  Hueckel 
[49,50]; 

b.  Gradient  Operator 

This  approach  has  been  studied  by  Duda  and  Hart  [57];  Rosenfeld  [61 3; 
Rosenfeld  and  Thurston  [50,59];  Rosenfeld  and  Troy  [68];  Rosenfeld  and 
Thomas  [Go];  Persoon  [673;  Wechsler  [112];  and  Thompson  [85]; 

c.  Boundary-Search 

This  approach  investigated  by  Rosenfeld  [52];  and  Kelley  [53l;  and 

d.  Line-Fi tting 

This  approach  developed  by  Hough  [54];  Duda  and  Hart  [55];  and 
Pavlidis  [56]. 

Host  of  the  above  techniques  are  statistical.  Hone  of  them  are 
based  on  the  syntactic  approach.  Hone  of  them  have  explicitly  used  struc- 
tural and  contextual  information.  These  statistical  techniques  suffer 
from  costly  computer  processing  time.  An  image  often  exhibits  a 
hierarchical  structure.  Therefore,  image  segmentation  can  be  approached 
by  the  syntactic  method.  A syntactic  method  based  on  the  finite  state 
(string)  grammar  approach  has  been  developed  for  image  recognition  in 
Chapter  3.  A syntactic  method  based  on  the  tree  grammar  approach  has 
been  developed  for  iamge  segmentation  in  Chapter  4.  ft  is  desirable  to 
make  syntactic  algorithms  useful  in  real-world  applications.  Thus,  these 
syntactic  algorithms  have  all  been  tested  on  real-world  data  such  as 
satellite  images,  aerophotographic  images,  and  infrared  images.  The  ex- 
perimental computer  results  show  that  these  syntactic  algorithms  are 
useful  for  image  recognition  and  segmentation. 
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1.2  DESIGN  FOR  A SPECIAL  COMPUTER  ARCHITECTURE  FOR  IMAGE  PROCESSING 

Previous  designs  for  special  computer  architecture  for  image 
processing  basically  fall  Into  two  categories:  bit-plane  processing  and 
distributed  processing. 

a.  Bit-Plane  Processing 

The  bit-plane  processing  approach  performs  the  arithmetic  computa- 
tion on  image  points  which  are  stored  in  Boolean  bit  planes.  The  special- 
purpose  computers  developed  by  this  approach  are  the  Illinois  pattern 
recognition  computer  (ILLIAC  III)  [87,100],  the  Digital  Parallel  Processor 
(DPP)  [90,91],  the  Cellular  Logic  Image  Processor  (CLIP  b)  [92],  and  the 
Parallel  Picture  Processing  Machine  (PPM)  [88,89]. 

b.  Distributed  Processing 

In  the  distributed  computing  approach,  the  configuration  of  the 
processors  forms  an  architecture  such  that  the  computational  load  is  shared 
by  these  processors  through  software  and/or  hardware  control . The  special 
purpose  computers  that  have  been  built  utilizing  this  approach  are  the 
Flexible  Processor  by  Control  Data  Corporation  [97] » Toshiba  Picture 
Processing  System  (TOSPICS)  [93,116]  by  Toshiba  Corporation,  and  STARAN 
computer  by  Goodyear  Aerospace  Corporation  [108,109]. 

A weakness  of  all  the  above  special  computers  for  image  processing 
is  t.'-at  the  computer  systems  are  not  reconf igurable,  that  is,  the  computer 
car.  only  operate  in  one  of  the  four  modes:  SISD  (single  instruction 
stream  single  data  stream),  MISD  (multiple  instruction  stream  single  data 
stream),  SIMD  (single  instruction  stream  multiple  data  stream),  or  MIMD 
(multiple  instruction  stream  multiple  data  stream).  Because  of  the  great 
variety  of  sensor  types,  and  the  many  applications,  image  processing 
algorithms  require  that  the  computer  system  be  reconf  igurable.  Therefore, 
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a design  for  a special  computer  for  image  processing  which  Is  reconflgur- 
able  is  proposed  in  Chapter  5. 

1.3  RESEARCH  SUMMARY 

This  research  deals  with  three  areas:  syntactic  pattern  recognition; 
information  extraction  and  image  understanding;  and  special  computer 
architecture  for  image  processing. 

In  the  area  of  syntactic  pattern  recognition,  the  finite-state 
string  grammar  approach  used  for  image  recognition  is  presented  in  Chapter 
3.  Two  syntactic  methods  have  been  developed.  One  is  a syntax-directed 
method  and  the  other  is  a syntax-controlled  method.  The  syntax-directed 
method  uses  a set  of  templates  as  a recognizer.  A deterministic  finite- 
state  automaton  is  used  as  a recognizer  In  the  syntax-controlled  method. 

An  interactive  grammatical  inference  procedure  was  devised  for  the  syntax- 
directed  method.  A k-tail  finite-state  grammatical  inference  procedure 
[20 1 , which  is  a procedure  to  minimize  the  number  of  states  by  equivalence 
partitioning  based  on  the  length  "k"  of  the  derivatives  of  the  sample 
patterns,  is  used  ir.  a fully  computer  automated  procedure  for  the  syntax 
controlled  method.  A comparative  study  on  the  syntax-directed  (commonly 
known  as  template  matching)  and  syntax-controlled  methods  were  undertaken. 
This  led  to  a decision  in  favor  of  the  syntax-controlled  method.  The 
performances  of  the  recognition  by  the  different  finite  state  grammars, 
which  are  inferred  by  the  k-tail  inference  procedure  with  different 
values  of  k,  were  studied.  This  revealed  that  the  grammar  inferred  by  a 
high  value  of  k is  more  precise  in  characterizing  the  syntactic  patterns 
than  that  inferred  by  a low  value  of  k,  but  the  computer  processing  time 
for  the  recognition  by  the  corresponding  finite  state  automaton  increases. 
These  syntactic  methods  have  been  implemented  and  applied  to  the 


0 


recognition  of  highways,  rivers,  bridges,  and  commercial/industrisl  areas 
from  satellite  images.  The  finite-state  string  grammar  has  been  shown  to 
be- able  to  characterize  the  structure  of  highways  and  rivers.  The  com- 
puter results  were  accurate  even  though  the  resolution  of  the  Input 
image  was  low  due  to  being  collected  by  satellite  at  a very  high  altitude 
(approximately  5/0  miles)  [82]. 

In  the  area  of  information  extraction  and  image  understanding,  the 
objective  of  the  research  was  to  achieve  a better  understanding  of  image 
structure  and  to  use  this  knowledge  to  develop  a technique  for  image 
analysis  and  automatic  information  extraction.  The  results  obtained  from 
applying  syntactic  pattern  recognition  to  satellite  images  cf  highways, 
rivers,  bridges,  and  commercial/industria]  areas  are  useful  for  image 
analysis  and  relevant  to  military  applications.  The  automated  methods 
of  extracting  such  information  as  position  coordinates  and  lengths  of 
bridges  and  centers  and  sizes  of  a commercial/industrial  areas  as  pre- 
sented in  Chapter  3 provides  a high  level  understanding  of  imagery  by  com- 
puter automation.  The  syntactic  image  segmentation  algorithm  presented 
In  Chapter  4 incorporates  textural  discrimination  and  boundary  structure 
analysis.  The  tree  grammar  approach  is  applied.  A tree  transformational 
grammar  and  its  inference  procedure  is  introduced  to  reduce  the  noise  and 
irregularities  in  patterns.  The  syntactic  image  segmentation  algorithm 
was  applied  to  tactical  target  detection  from  infrared  images. 

In  the  area  of  special  computer  architecture  for  image  processing, 
several  previously  proposed  special  computers  for  Image  processing  were 
reviewed.  The  proposed  special  computer  architecture  in  Chapter  5 was 
designed  using  a distributed  computing  approach.  This  computer  is  com- 
prised of  a Parallel  Processor  (PP)  and  a Sequential  Arithmetic 
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Processor  (SAP).  There  are  two  major  features  new  to  the  field  of  special 
computer  architecture  for  image  processing.  They  are  a reconfigurable 
capability,  and  a special  method  of  parallel  ism.  These  contribute  a high 
flexibility  and  a high  performance  capability  for  image  processing  to 
the  proposed  computer  architecture.  The  reconfigurable  capability  en- 
ables the  proposed  computer  to  satisfy  the  large  variety  of  applications 
in  image  processing.  This  capability  is  obtatned  through  the  Control 
Unit  of  Parallel  Processor  (CUPP)  which  reconfigures  the  parallel 
processor  between  the  SIMD  mode  and  the  MIMD  mode.  The  parallelism  of 
the  task  is  exploited  by  the  parallel  processor  to  obtain  high  speed 
performance.  At  the  same  time  the  operations  of  the  sequential  arith- 
metic processor  are  pipelined  to  the  parallel  processor  under  program 
control  in  those  tasks  which  can  be  decomposed  Into  pipeline  processing. 
Therefore,  the  exploitation  of  parallelism  results  In  parallelism  and 
pipelining  simultaenously.  Thus,  the  computer  architecture  achieves  high 
flexibility  and  performance. 
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CHAPTER  2 

PREVIOUS  WORK 

2.1  PREVIOUS  RESEARCH  WORK  1H  IMAGE  SEGMENTATION 

As  mentioned  In  Chapter  1,  previous  research  in  the  field  of  image 
segmentation  falls  into  three  major  categories;  characteristics  thres- 
holding, region  extraction,  and  edge  detection. 

2.1.1  Characteristics  Thresholding  Methods 

The  whole  field  of  image  processing  has  became  possible  because  of 
the  development  of  the  ability  to  "digitize"  a picture  or  image;  that 
is,  the  picture  is  digitized  into  a matrix  of  n by  n pixels  (picture 
elements).  Then,  each  pixel  is  given  a grey  level  value  corresponding 
to  the  amount  of  light  it  transmits.  These  grey  levels  lend  themselves 
to  mathematical  manipulation,  and  functions  can  be  written  Involving 
them.  The  first  task  is  to  find  characteristic  functions  of  grey  levels. 
The  second  is  to  find  a threshold  in  such  a function  that  will  make  a 
significant  division  in  the  type  of  pixels.  There  are  two  basic 
approaches  to  thresholding  the  characteristic  function  of  grey  levels 
for  image  segmentation;  the  statistical  and  the  structural, 
a.  Statistical  Approach 

In  this  approach,  a picture  is  divided  Into  different  regions  by 
thresholding  the  value  of  an  appropriate  local  picture  property.  For 
example,  all  physical  objects  have  a constant  reflectance  over  their 
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surfaces.  Therefore,  an  object  of  uniform  grey  level  is  considered  to 
be  a homogeneous  object.  Zucker,  Rosenfeld,  and  Davis  [26]  proposed 
that  a good  method  of  segmenting  a picture  Into  regions  of  uniform  grey 
levels  would  be  to  examine  the  histogram  of  the  grey  levels  in  the 
picture.  Instead  of  measuring  the  grey  level  of  each  picture  point,  a 
set  of  spot  detectors  having  a range  of  sines  is  used  to  provide  local 
property  values  upon  which  to  base  the  segmentation.  There  is  one 
problem  in  this  technique;  that  is,  that  due  to  overlap  of  the  spot 
detector  In  the  measurement  of  each  point,  sometimes  the  histogram 
valley,  necessary  to  threshold  the  picture  could  not  be  found.  So  non- 
maximun  feature  values  were  suppressed  over  an  area  corresponding  to 
the  receptive  field  of  each  spot  detector;  that  is,  a spot  value  was 
ignored  if  a larger  value  existed  at  some  point  In  its  receptive  field. 
The  threshold  for  segmentation  was  selected  as  the  lowest  point  between 
two  peaks  in  the  histogram.  But  there  are  only  limited  results  on 
bimodel  cases  in  Zucker,  Rosenfeld,  and  Davis’s  [26]  paper.  Further 
complicated  experiments,  such  as  the  multimodel  cases  should  be  studied 
in  order  to  show  the  generality  of  this  technique. 

In  some  cases,  where  the  black  and  white  dots  occur  in  both  the 
significant  regions  and  in  the  background  (e.g..  In  the  cases  where 
the  probability  of  occurrence  of  a black  dot  Is  0.6  in  the  significant 
region  and  0.4  in  background),  the  method  in  [26]  will  not  work  because 
the  threshold  cannot  be  found.  Davis,  Rosenfeld,  snd  Weszka  [29] 
applied  local  averaging  of  grey  level  values  to  every  point  of  the 
picture.  Then  the  histogram  was  built  by  the  average  grey  levels  of 
the  picture.  But  the  method  will  not  work  If  the  picture  also  contains 
other  adjacent  regions  In  which  the  average  grey  levels  are  higher  and 


12 


lower  than  that  of  the  initial  region.  Weszka,  Nagel,  and  Rosenfeld 
[30]  apv'ied  the  Laplacian  operator  to  determine  points  that  lie  on  or 
near  the  edges  of  objects  to  try  to  get  around  this  problem.  In  work- 
ing with  a grey  level  histogram,  the  peaks  are  sometimes  very  unequal 
In  size  and  the  valley  is  broad.  Here,  the  thresholding  Is  difficult. 

Ohlander  [68]  used  multiple  sources  of  data  and  the  threshold 
operation  based  on  histograms.  Thus,  there  exists  the  option  of 
examining  nine  histograms  from  these  sources  to  dt termine  the  most 
sharply  defined  feature.  Thresholding  on  limits  provided  by  the 
minima  bounding  the  best  peak  Jn  the  histogram  will  furnish  clusters 
of  points  which  are  uniform  for  the  given  feature.  Then  these  regions 
are  extracted.  But,  the  number  of  sensory  parameters  and  variety  of 
picture  operations  require,  for  this  system,  large  amounts  of  storage 
space  and  heavy  expenditures  of  computational  time.  In  this  system, 
the  control  of  parameters  is  determined  by  human  interaction.  Even 

with  human  interaction  and  9 hours  of  CPU  time,  the  heavy  input  and  out- 
put requirements  increases  the  real  time  processing  to  18  hours  or  more 
to  process  a scene  of  600x800  points  on  a PDP  10  computer.  And  as  the 
amount  of  noise  in  the  information  becomes  greater  and  greater,  the 
resolution  of  this  system  becomes  less  and  less. 

Carlton  and  Mitchell  [86]  used  texture  and  grey  level  information 
for  image  segmentation.  This  technique  uses  a texture  measure  that 
counts  the  number  of  local  extrema  in  a window  centered  at  each  pixel. 
This  results  In  an  Intermediate  grey  level  picture  representation  of  a 
texture  property.  These  intermediate  pictures  are  used  to  derive 
starting  points  in  each  region  to  be  segmented.  The  segmentation  is 
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completed  by  assigning  each  pixel  to  a starting  point  using  a distance 
criteria.  For  this  technique,  there  are  several  thresholds  which  must 
be  set  to  make  it  operate;  namely,  extrema  size,  windcw  sizes,  and 
distance  criteria.  If  the  Input  data  is  not  quite  homogeneous,  the 
optimal  set  of  thresholds  needs  to  be  found. 

Wall,  Klinger  and  Castleman  [27]  and  Wall  [28]  proposed  five  models 
to  represent  objects  in  image  analysis  and  their  corresponding  histograms. 
The  five  models  are  the  ideal  object  image  function,  the  truncated 
wedge  function,  the  circular  Gaussian  function,  the  Gaussian  through 
function,  and  the  Gaussian  edge  function.  The  ideal  object  image 
function  I (x,y)  is  a function  having  constant  grey  level  1^,  inside  an 
arbitrary  set,  s,  and  zero  outside.  S is  a simple,  connected  subset  of 
the  region  with  a boundary  of  arbitrary  shape.  If  (x,y)GS,  I (x.y)  - lm; 
if  (x,y)0S,  1 (x,y)  = 0.  The  truncated  wedge  function  is  a function 
having  constant  grey  level,  1^,  and  slope,  b,  from  the  constant  value 
of  grey  level  to  zero  value  of  grey  level.  The  circular  Gaussian 
function  of  object  image  is  defined  by 

-[(x-U x)2+(y-U  )~]/202  ; (x,y)6d 

l(x,y)  = lne  y 


I (x,y)  = 0 ; (x,y)0d 
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constants  andZ>  3x.  The  Gaussian  through  function  is  defined  by 
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L.  are  arbitrary  constants  andZ>  3 a.  The  Gaussian  edge  function  Is 
generated  by  placing  one-half  of  a circular  Gaussian  Image  function  on 
each  end  of  the  Gaussian  through  function. 

For  the  extension  of  grey  level,  Yachlda  and  Tsujl  [33l  found 
uniform  color  regions  by  utilizing  color  information.  So,  color  in- 
formation is  helpful  in  image  segmentation. 

b.  Structural  Approach 

In  the  structural  approach  to  characteristics  thresholding,  the 
uniformity  of  complex  Image  properties  such  as  size,  shape,  or  arrange- 
ment of  subpatterns  in  a picture  is  examined. 

Tsuji  and  Tomita  [32],  and  Tomita  and  Yachlda  [31]  described  a 
method  for  dividing  the  input  scene  Into  regions  by  thresholding  the 
histograms  of  the  grey  level  values  of  several  structural  descriptors. 
For  the  purpose  of  saving  memory  space  and  computing  time,  several 
specific  descriptors  were  selected.  They  are  shape,  size,  position, 
and  density.  The  size  descriptor  specifies  the  area  and  perimeter  of 
each  unit  region.  This  position  descriptor  gives  the  two  dimensional 
coordinates  of  the  center  of  gravity  of  the  unit  region.  The  shape 
descriptor  is  selected  in  such  a way  that  It  gives  rough  information 
about  the  shape  and  is  not  sensitive  to  sampling  noise  In  the  process 
of  digitizing  the  picture.  A region  A In  a given  set,  s,  has  a density 
descriptor,  Ds,  whose  value  is  the  minimum  distance  from  A to  other 
regions  In  s.  The  partitioning  procedure  works  as  following:  first, 
the  picture  descriptors  are  evaluated  and  their  list  is  constructed. 
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Second,  the  histograms  of  the  values  of  the  descriptors  are  computed 
and  a filtering  technique  smooths  the  small  peaks  that  lid  close  to 
each  other.  Third,  a supervisor  selects  the  most  promising  descriptors 
for  classifying  the  regions  Into  groups.  The  most  promising  descriptors 
are  those  descriptors  whose  histograms  have  deep  valleys.  The  super- 
visor puts  the  thresholds  at  the  bottoms  of  the  valleys.  Fourth,  the 
density  descriptors  of  the  members  of  each  group  are  evaluated.  If 
some  members  have  density  descriptors  whose  values  are  larger  than  a 
predetermined  threshold  value,  they  are  excluded  from  the  groups  as 
isolated  regions.  Finally,  a boundary  test  is  done  to  check  whether 
or  not  any  regions  In  the  group  of  isolated  elements  are  located  between 
tuo  regions  or  surrounded  by  a region.  They  are  merged  with  the  region 
if  only  one  region  touches  them.  Thus,  the  structural  analysis  »s 
finished  and  the  goal,  partition  of  the  picture.  Is  achieved. 

2.1.2  Region  Extraction  Methods 

These  methods  utilize  extraction  of  regions  to  segment  the  pic- 
ture. There  are  three  approaches  to  region  extraction:  merging, 
dividing,  a combination  of  merging  and  dividing. 

a.  Region  Merging  Approach 

The  region  merging  approach  begins  with  a well  formed  partition 
(e.g.,  the  picture  consists  of  n square  pixels  and  the  pixel  is  |x(  in 
size),  and  processes  It  by  merging  adjacent  elements  together  that  are 
found  to  be  similar  in  certain  characteristics.  There  are  five  types 
of  region  merging:  statistical,  linguistic,  decision-theoretic, 
•'relaxation”,  and  interpretation-guided. 
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(!)  Statistical  Region  Merging 

Here,  the  merging  criteria  are  based  on  certain  statistical  charac- 
teristics. Brice  and  Fennema  [3*0  use  unit  regions  as  basic  data  and 
process  them  by  successive  merging  of  the  unit  region  toward  a final 
image  partition.  Similar  approaches  used  by  Strong  and  Rosenfeld  [35], 
Harlow  and  Eisenbers  [36],  and  Rodd  [37].  Bajcsy  [39]  uses  two  types 
of  merging  of  logic  sequences;  from  left  to  right,  and  from  right  to 
left.  Kettig  [33]  uses  a similar  method  on  multispectral  remotely 
sensed  imagery  data.  His  experimental  results  show  that  this  approach 
suffers  from  slow  computer  processing  tine.  Gupta  and  Wintz  [4C]  pro- 
posed a merging  sequence  which  expands  horizontally  and  vertically  by 
absorbing  more  and  more  pixels  until  It  reaches  nature  boundaries. 

(ii)  Linguistic  Region  Merging 

Since  most  grammars  are  useful  in  analyzing  a one  dimensional 
string,  Tsuji  and  Fujiwara  £41 3 proposed  applying  sequentially  two 
grammars  of  one  dimensional  strings  for  picture  segmentation.  The 
picture  primitives  are  line,  curve,  edge  or  undefined  segments.  The 
system  works  as  follows:  the  first  stage  of  processing  is  the  search 
for  line  segments.  Then  a line  fitter  tries  to  connect  the  line  seg- 
ments. The  horizontal  grammar  Is  constructed  manually  in  that  a hori- 
zontal parser  gives  a label  to  each  segment,  and  examines  the  horizontal 
contexts  in  order  to  join  several  segments  into  a new  longer  segment. 

The  results  are  the  set  of  picture  sentences  of  horizontal  scan.  Based 
on  the  concept  of  coupling,  a set  of  grammar  rules  has  been  written  as 
vertical  grammar.  The  vertical  parser  analyzes  the  vertical  contexts 
of  symbols  in  picture  segments  and  generates  region  sentences  which 
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give  the  characteristics  of  all  surfaces  In  the  picture.  The  final 
step  of  vertical  parsing  Is  to  combine  one  region  with  another.  This 
approach  works  fairly  well  on  some  artificial  block  type  of  images, 
but  fails  in  classifying  scenes  that  have  high  lights,  shadows,  and 
different  textures. 

(lil)  Decis  ion-Theoretic  Region  Merging 

This  approach  applies  Bayesian  decision  rule  techniques  and  uses 
problem-dependent  information  (semantics)  to  solve  the  picture  segmen- 
tation problem.  Yakimovsky  and  Feldman  [*l2]  and  Yakimovsky  [i»3]  present 
a theoretical  framework  for  a system  which  incorporates  neighborhood 
information  in  a region  analyzer.  The  two  central  ideas  are  the  use  of 
a utility  function  to  measure  the  value  of  various  alternatives,  and  an 
optimality  theorem.  A disadvantage  in  the  approach  is  the  assumption 
that  the  interpretation  of  a region  depends  only  upon  adjacent  regions. 
The  choice  of  local  measurements  around  each  point  is,  of  course,  a 
crucial  factor  but  interpretation  can  sometimes  depend  on  a region 
quite  far  away.  Also  it  seems  that  the  computer's  "learning"  procedure 
needs  to  be  refined  in  this  approach. 

(iv)  "Relaxation"  Region  Merging 

Another  approach  to  region  merging  is  called  the  "relaxation" 
process  proposed  by  Rosenfeld,  et  al  [78].  This  process  has  been  applied 
to  scene  labeling,  line  enhancement  [80] , and  template  matching.  The 
relaxation  process,  also  called  iterative  probabilistic  process,  first 
estimates  for  each  point.  P,  the  probability,  P.,  that  it  belongs  to 
each  of  the  possible  classes.  Then  the  P.  is  increased  if  supporting 
evidence  is  found  for  it  or  decreased  where  contradictory  evidence  is 
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found.  When  for  a given  point,  P,  only  one  class  has  a high  probability, 
then  it  is  classified  as  that  class.  In  this  approach,  the  estimated 
initial  probability,  P.,  for  each  point  would  effect  the  converging  time 
ior  iteraction.  The  computer  processing  time  can  be  costly  for  this 
method  because  of  the  large  number  of  iteraction  needed. 

(v)  Interpretation-Guided  Region  Merging 

Tenenbaum  and  Barrow  [79]  proposed  the  IGS  (Interpretation  Guided 
Segmentation)  approach  to  region  merging.  This  technique  relates  human 
knowledge  to  a certain  set  of  rules  and  iteractively  processes  these 
rules  until  it  achieves  a final  segmentation  result.  Finding  a set  of 
rules  which  effectively  describes  the  knowledge  of  the  scene  is  impor- 
tant. 8ecause  the  rules  have  to  be  iteractively  processed,  the  computer 
processing  time  is  large. 

b.  Region  Dividing  Approach 

The  region  dividing  approach  begins  with  dividing  the  picture  into 
several  parts  then  subdividing  each  part  until  the  partitioning  satis- 
fies certain  stopping  criteria.  Robertson  [44]  designed  his  partition 
algorithm  on  the  assumption  that  a region  contains  a boundary.  The 
homogeneous  property  that  he  considers  is  the  mean  vector  of  brightness 
functions  of  the  mul tispectral  remotely  sensed  image. 

Klinger  [45]  suggests  a partition  idea  through  the  refinement  of 
objects  into  four  equal  quadrants  such  as,  northwest,  northeast,  south- 
west, and  southeast  quadrants.  He  also  [69]  applies  a regular  decom- 
position to  divide  the  picture  area  into  successively  smaller  quadrants. 
The  concept  of  regular  decomposition  Is,  first  to  represent  a digitized 
picture  consisting  of  spatial  subsets  of  different  sizes  marked  either 


•/fT 


J 


19 


"Informative  for  scene  description"  or  "non- Informative"  and  second,  to 
discard  picture  elements  (pixels)  that  belong  to  "non-informative"  sub- 
sets. Initially,  the  entire  digitized  picture  is  a quadrant.  If 
nothing  informative  is  contained,  such  a quadrant  may  be  entirely  eli- 
minated from  the  data  structure.  If  a large  amount  of  information  is 
found  in  the  quadrant,  the  quadrant  should  be  saved.  If  the  decomposi- 
tion algorithm  fails  to  make  a decision  about  a picture  quadrant,  it 
is  subdivided  and  each  of  the  four  quadrants  is  processed  by  the  same 
procedure.  Thus,  the  regular  decomposition  is  a process  of  searching 
for  picture  areas  where  there  is  "informative"  data  present.  This 
method  is  used  to  build  computer-searchable  data  representation  of  an 
image. 

c.  Combination  of  Merging  and  Dividing 

This  approach  to  region  extraction  starts  from  an  arbitrary  parti- 
tion and  dynamically  determines  when  to  merge  or  divide.  Horowitz  and 
Pavlidis  [46]  used  this  idea  fn  devising  their  image  segmentation  by  a 
directed  spi l t-and-merge  procedure.  They  have  also  refined  their 
algorithm  by  applying  a grouping  algorithm  which  combines  two  adjacent 
regions,  provided  these  initial  regions  satisfy  a given  property.  It 
eliminates  small  regions  which  are  due  to  noise  or  to  transitions 
between  large  regions.  For  the  purpose  of  identifying  multiple  con- 
nected regions  corresponding  to  cut  notes  in  the  picture,  Horowitz  and 
Pavlidis  [70]  used  graph  analysis  to  identify  these  regions.  But  the 
a priori  information  of  the  given  property  that  they  use  is  the  draw- 
back to  their  algorithms. 
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2.1.3  Edge  Detection  Methods 

Another  method  for  image  segmentation  is  finding  the  boundaries 
between  regions  in  order  to  extract  the  homogeneous  regions.  There  are 
several  approaches;  such  as,  template  matching,  gradient  operator, 
boundary  searching  and  line  fitting. 

a.  Template  Matching 

Template  matching  can  be  done  in  a dec  is  ion- theoretic  way.  The 
decision  problem  is  to  compute  the  probability  that  a line  representing 
a real  edge  is  centered  in  some  iong  narrow  area.  Griffith  £47,^8 ,573 
discusses  the  optimal  use  of  intensity  information  to  detect  edges  in  a 
block  world.  Hueckel  [43]  proposes  a method  by  asking  what  edge  element 
will  best  fit  the  intensities  in  a given  region.  An  extension  of  the 
foregoing  techniques  to  detect  line  and  edge-line  is  presented  In 
Hueckel  [50].  This  approach  iscapableof  telling  the  exact  orientation 
of  the  line  segment. 

b.  Boundary  Operator 

The  gradient  boundary  operator  is  defined  as 

Vg(X,y)  t*  p-V 

v ' 8x  x 3y  y 
and  i Vg (x,y)  | = l(|^)2  + (fy-)2]1/2. 

If  the  picture  is  noisy,  then  smoothing  techniques  snould  be  employed 
before  applying  the  gradient  operator  according  to  [57].  Rosenfeld  [61] 
indicates  that  when  there  Is  noise  incorporated  into  the  smoothing  pro- 
cedure, the  larger  the  size  of  the  neighborhood,  the  less  precisely  is 
the  edge  located.  Procedures  for  detecting  abrupt  changes  in  average 
grey  level  value  in  order  to  locate  the  edges,  are  presented  in  Rosen- 
feld and  Thurston  [58,59]*  Rosenfeld  and  Troy  [66]  and  Rosenfeld  and 
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Thonas  [60]  apply  this  concept  to  detect  the  texture  edges  where  the 
two  regions  differ  with  respect  to  the  average  value  of  some  local 
properties.  Recently,  Persoon  [67]  devised  a new  gradient  operator 
algorithm  for  edge  detection  performing  experiments  on  rib  extraction 
from  chest  x-ray  pictures. 

In  March  of  1977,  Thompson  [85]  applied  the  modified  Roberts  cross 
operator  for  textural  boundary  detection.  The  Roberts  cross  operator 
is  defined  as  R < i , j ) = |P (I ,J)  - P(i+l,j+l)|  + |P(i+l,j)  - P(i , j+1 ) { . 

In  [85],  the  textural  boundary  operator  is  proposed  as  T = D(a,d)  + 
D(b,c).  D(a,d)  is  the  computed  texture  dissimilarity  between  region  a 
and  d.  Regions  a,  b,  c and  d are  the  quadrants  of  northwest,  northeast, 
southwest,  and  southeast  respectively.  The  edge  finding  method  con- 
sists of  two  stages.  First,  a map  is  produced  by  applying  the  textural 
boundary  operator  to  selected  points  in  an  image.  A second  edge  map 
is  then  provided  by  smearing  each  point  In  the  first  map  along  the 
direction  of  edge  orientation. 

c.  Boundary  Searching 

This  approach  detects  the  boundary  by  means  of  applying  an  operator 
to  search  the  picture  to  locate  the  boundary  between  regions.  One  of 
the  techniques  called  "the  contour  following  technique"  is  an  applica- 
tion of  the  idea  that  previous  knowledge  about  the  existence  of  an  edge 
from  the  operator  might  give  us  a good  prediction  of  the  location  of 
the  next  edge  element  [52],  In  [53],  the  planning  strategy  was  applied 
to  the  contour  following  search.  The  technique  of  contour  following 
with  planning  strategy  should  be  quite  good,  as  it  aqplies  knowledge 
from  previous  small  areas.  But  It  fails  when  the  parts  of  the  picture 
are  independent  and  uncorrclated. 
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d.  Line  Fitting 

Hough  [5^3  suggested  the  use  of  a set  of  parameters  to  fit.  a group 
of,  points  with  line  segments.  Duda  and  Hart  [553  indicated  that  the  use 
of  angle-radius  rather  than  slope-intercept  parameters  will  simplify  the 
computation  in  Hough's  work.  Pavlidis  [56]  proposed  numerical  functional 
approximation  to  fit  the  boindary.  Several  algorithms  of  boundary  detec- 
tion [62,63,6^,653  are  of  this  category.  These  techniques  can  contribute 
to  the  segmentation  of  a picture.  But,  sometimes  structure  of  the 
boundary  of  a picture  is  complicated  and  the  line  fitting  approach  be- 
comes inadequate. 

2.2  PREVIOUS  WORK  IN  SYNTACTIC  PATTErN  RECOGNITION 

In  syntactic  pattern  recognition,  a "grammar"  is  used  to  characterize 
a set  of  patterns.  The  syntactic  method  has  the  power  of  describing  and 
classifying  patterns.  For  the  description  of  a pattern,  a pattern  is 
represented  by  a sentence  in  a language  which  is  specified  by  a grammar 
[2,5]..  This  language  provides  the  structural  description  of  patterns  by 
a set  of  pattern  primitives  and  their  relational  rules.  The  recognition 
of  a pattern  is  accomplished  by  a syntax  analysis  according  to  the 
grammar.  A block  diagram  of  a syntactic  pattern  recognition  system  is 
shown  in  Figure  2.1.  The  upper  part  of  the  diagram  is  the  recognition 
part  and  the  lower  part  Is  the  analysis  part.  For  the  purpose  of  de- 
scribing and  recognizing  the  pictorial  patterns,  various  two-dimensional 
grammars  called  tree  grammars,  web  grammars,  and  graph  grammars  have  been 
developed  and  applied  to  syntactic  pattern  recognition.  A good  survey  of 
the  early  work  in  the  field  of  syntactic  pattern  recognition  was  written 
by  Fu  [2].  More  recent  surveys  are  found  in  [4,5,6»12]. 
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Fig.  2.1  Block  diagram  of  syntactic,  pattern  recognition  system 


2h 

Previous  research  v/ork  in  this  field  falls  into  three  major  areas;  namely; 

grammatical  inference,  syntactic  analysis,  and  applications. 

s 

2.2.1  Grammatical  Inference 

Fu  and  Booth  wrote  an  intensive  survey  on  grammatical  inference  in 

[20].  The  grammatical  inference  is  part  of  the  analysis  for  syntactic 

pattern  recognition  shown  in  Figure  2.1.  In  order  to  realistically  de- 

scribe  a class  of  patterns  under  study,  the  grammar  used  in  syntactic 

pattern  recognition  is  hoped  to  be  directly  inferred  from  a set  of  sample 

patterns.  Figure  2.2  shows  the  general  model  of  the  grammatical  inference 

process  [2],  A source  generates  sentences  or  patterns  of  form  x.  = a.  , 

' *1 

...,  a.  where  the  symbolsa.  are  elements  from  a finite  set  called 
n 

the  set  of  terminal  symbols.  These  sentences  are  assumed  to  possess  some 
unique  structural  features  which  are  characterized  by  a grammar,  G,  which 
can  be  used  to  model  the  source.  All  of  the  sentences  which  can  be  gen- 
erated by  the  source  are  contained  in  the  set  L(G),  the  language  generat- 
ed by  G,  while  all  of  the  sentences  which  cannot  be  generated  are  contained 
in  the  complement  set  L (g)  . An  observer  is  given  a finite  set  s+ of  sen- 
tences which  are  from  L(G)  and  another  finite  set  s of  sentences  from 
lTgF.  Using  this  information  the  observer  infers  the  syntactic  rates  of 
the  unknown  grammar  G.  The  sentences  which  belong  to  s+are  defined  by 
the  properties  of  G.  These  sentences  from  s are  also  input  to  the  ob- 
server. In  section  2.3  of  this  chapter,  more  details  on  grammatical  in- 
ference algorithms  are  presented.  A tree  grammar  inference  algorithm  is 
described,  and  an  other  grammatical  inference  algorithm  for  a tree  trans- 
formational grammar  Is  proposed. 


Fig.  2.2  General  model  of  grammatical  inference  process 
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2.2.2  Syntactic  Analysis 

After  a grammar  is  constructed  to  generate  a language  which  would 
describe  the  patterns  under  study,  the  next  step  is  to  design  a recogniz- 
er that  will  recognize  the  patterns  according  to  the  grammar.  The  recog- 
nizer designed  for  a particular  grammar  will  recognize  only  the  patterns 
in  the  class  corresponding  to  this  grammar.  If  the  grammar  is  finite, 
the  syntactic  analysis  is  done  by  the  finite  state  automata  [2],  A 
recent  development  in  this  area  is  error-correcting  syntactic  analysts 
[16,19,22],  In  some  applications,  a certain  amount  of  uncertainty  exists 
in  the  process.  In  order  to  describe  noisy  and  distorted  patterns,  the 
use  of  transformational  grammars,  stochastic  languages,  and  approximation 

have  been  suggested  [2,5,23].  For  the  recognition  of  noisy  and  distorted 
patterns,  error-correcting  syntactic  analyzers  have  been  proposed.  Fung 
and  Fu  [19,22]  used  the  maximum  likelihood  decision  criterion  as  decision 
rule  to  resolve  ambiguities  for  stochastic  language.  The  basic  approach 
is  to  use  the  concept  of  transformations  of  strings.  The  error-correct- 
ing parser  first  induces  error  transformations  into  the  original  grammar 
to  give  a covering  grammar  whose  language  is  universal.  To  search  for 
the  string  that  satisfies  the  decision  rules,  the  parsing  algorithm  is  a 
conventional  parser  with  a provision  added  for  bookkeeping  of  the  number 
of  transformations  used.  The  probabilistic  models  and  stochastic  error- 
correcting  parsing  techniques  were  applied  to  the  recognition  of  noisy 
patterns. 

,:Tree"  system  approach  was  first  introduced  to  syntactic  pattern 
recognition  by  Fu  and  Bhargava  [13].  In  [16],  the  error-correcting  tree 
automata  wa s proposed.  Unlike  the  string  case,  where  only  the  relation 
between  symbols  Is  left-right  concatenation,  a tree  structure  would 
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become  deformed  by  deletion  or  insertion  errors.  The  structure-preserved 
error-correcting  tree  automata  takes  only  substitution  errors  into  con- 
sideration. A generalized  error-correcting  tree  automaton  consists  of 
five  types  of  error  transformations;  namely,  substitution,  stretch, 
branch,  splits  and  deletion.  The  distance  between  two  trees  Is  the 
least-cost  sequence  of  error  transformations  needed  to  transform  one 
tree  to  the  other.  Tree  distance  is  measurable  between  trees  of  different 
structures.  Based  on  the  measurements,  the  error-correcting  parser 
accepts  structurally  distorted  as  well  as  node-mislabeled  trees  such  that 
their  minimum  distance  corrections  can  be  found  [163 . 

For  the  purpose  of  increasing  the  flexibility  of  the  syntactic 
method,  Fu  and  Lu  [17]  proposed  a clustering  procedure  for  syntactic  pat- 
terns. This  procedure  measures  the  distance  between  patterns  and 
establishes  a similarity  measure.  A similarity  measure  between  two  syn- 
tactic patterns  includes  the  similarities  of  both  their  structures  and 
primitives.  A riea rest-neighbor  recognition  rule  is  then  applied  with 
the  similarity  measures  and  a clustering  algorithm  is  proposed  for 
syntactic  patterns.  If  the  correct  classifications  of  pattern  samples 
are  known,  the  proposed  nearest  neighbor  recognition  rule  can  be  applied 
to  determine  the  classification  and  structural  description  of  an  un- 
known pattern.  When  the  correct  classification  of  pattern  samples  is 
unknown,  a non-supervised  procedure  must  be  used.  In  this  case,  the 
clustering  procedure  can  still  be  applied.  When  using  error-correcting 
parsers  in  cluster  analysis,  after  the  clustering  result  is  obtained, 
only  a conventional  non-error-correcting  parser  needs  to  be  implemented 
for  recognition.  The  flexibility  of  the  syntactic  method  is  greatly 
improved  by  this  clustering  procedure. 
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2.2.3  Appl icatlons 

Fu  and  Bhargava  [2]  applied  the  tree  system  approach  to  the  analysis 
of'Bubble  Chamber  films.  The  tree  system  approach  utilizes  a set  of 
grammars  to  describe  and  recognize  the  Bubble  Chamber  patterns,  (toayer 
and  Fu  [11]  applied  the  tree  system  approach  which  utilizes  sets  of 
grammars  to  describe  and  recognize  fingerprint  patterns.  The  tree  system 
approach  was  also  applied  to  LANDSAT  data  interpretation  which  analyzes 
the  processed  results  from  statistical  methods  to  improve  some  of  the 
data  interpretation  of  LANDSAT  images  [15].  Brayer  and  Fu  [1*»]  and  Fu 
[3,A]  approached  the  LANDSAT  data  interpretation  through  "web"  grammar 
analysis  which  utilizes  web  grammar  to  describe  the  contextual  and 
spatial  information.  A web  grammar  model  was  developed  and  used  to  im- 
prove the  accuracy  of  the  classification  and  to  find  some  new  classes. 

Pavlidis  [23, 9o]  applied  the  grammars,  which  are  characterized  by  a set 
of  primitives  corresponding  to  the  case  under  study,  to  shape  recogni- 
tion. His  applications  are  numeral  and  character  recognition,  and 
industrial  circuit  board  defect  detection. 

2.3  COMMENTS  ON  THE  PREVIOUS  RESEARCH  WORK 

Compared  with  the  techniques  in  the  "characteristics"  thresholding 
approach,  the  grey  level  histogram  provides  global  knowledge  about  the 
segmentation  of  a picture.  Concerning  the  computer  processing  time  in- 
volved In  image  segmentation,  the  grey  level  histogram  thresholding 
approach  is  quite  fast  because  the  mathematical  operator  manipulation  is 
much  simpler  than  gradient  or  Laplacian  thresholding.  But  in  the  histogram 
thresholding  approach,  where  there  are  small  regions,  they  show  up  in  the 
histogram  as  small  peaks.  Henc*.,  most  of  the  time  the  histogram  thres- 
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holding  approach  fails  to  segment  these  regions  of  the  image.  Another 
problem  is  the  case  In  which  two  or  more  different  objects  have  many 

overlapping  parts  in  the  histogram;  here  again  the  approach  fails  to 

s 

segment  the  Image.  Therefore,  techniques  similar  to  the  gradient  opera- 
tion, Laplacian  operation  or  modified  Laplaclan  operator  have  been  developed 
by  thresholding  some  local  property,  and  Ohlander  168]  tried  using  nine 
different  types  of  sensory  data  for  thresholding  histograms.  All  these 
approaches  suffer  from  heavy  expenditures  due  to  slow  computer  processing 
time.  After  this  examination  of  the  tresholding  of  histograms,  we  must 
consider  that  it  is  not  the  best  approach  for  image  segmentation. 

When  structures  of  patterns  or  textures  of  the  image  are  complex, 
the  characteristics  thresholding  method  by  structural  approach  is  useful. 
But,  the  discrimination  between  the  elementary  subpatterns  or  grains  of 
the  image  has  to  be  assumed  to  be  easily  obtainable  and  this  structural 
approach  needs  much  computer  processing  time. 

The  region  extraction  method  is  attractive  in  two  major  approaches; 
the  merging  approach  and  the  dividing  approach.  In  comparing  these  two 
approaches,  if  the  region  merging  approach  is  used  every  time  two  regions 
are  merged  together,  the  sample  statistics  can  be  calculated  simply  from 
the  staistlcs  of  those  two  regions  and  thus  the  processing  time  is  not 
so  large.  In  the  region  dividing  approach,  every  time  a region  is  divided, 
new  statistics  must  be  calculated  again  causing  large  processing  time. 

The  combinatipn  of  merging  and  dividing  is  considered  a combinatory  method, 
but  the  computer  processing  time  is  still  very  large. 

In  the  edge  detection  methods  for  image  segmentation,  if  boundaries 
between  regions  can  be  expressed  by  some  definite  form,  then  template 
matching  Is  suggested.  From  these  observations,  there  seems  to  be  dif- 
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ferent  appropriate  approaches  for  different  types  of  images.  That  is, 
ft  is  difficult  to  find  a general  method  for  all  Images.  Some  of  these 
techniques  calculate  second  order  statistics  and  some  of  them  need  planning 
strategy.  However,  all  of  these  techniques  require  a great  deal  of  com- 
puter processing  time,  in  conclusion,  we  consider  that  there  are  problems 
associated  with  all  of  the  previous  methods.  Therefore  in  Chapter  4, 
the  syntactic  image  segmentation  method  is  proposed.  Before  presenting 
the  image  segmentation  by  tree  grammar  (high  dimensional}  in  Chapter  4, 
the  image  recognition  by  finite  state  string  grammar  (one  dimensional) 
will  be  presented  in  Chapter  3. 
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CHAPTER  3 

IMAGE  RECOGNITION  BY  A STRING  GRAMMAR  APPROACH 

The  syntactic  approach  to  image  recognition  has  three  major  parts 
(Figure  3.1):  Preprocessor,  Syntactic  Analyzer,  and  Postprocessor  [10]. 
The  preprocessor  extracts  "primitives"  from  an  input  image  and  transforms 
the  image  to  "language"  sentences  which  are  the  inputs  for  the  syntactic 
analyzer.  This  process  usually  involves  a transformation  operation,  a 
threshold  operation,  and  an  averaging  operation  among  others.  The  syn- 
tactic analyzer  processes  the  input  image  by  a set  of  grammatical  rules 
Inferred  by  a grammatical  inference  algorithm.  This  procedure  is  in 
order  to  accept  those  patterns  which  can  be  generated  by  the  set  of  gram- 
matical rules,  and  to  reject  all  others.  The  postprocessor  sequentially 
executes  several  information  (semantic)  rules.  Those  objects  which  are 
related,  based  on  semantic  information,  are  recognized  from  the  image. 
Depending  on  the  structures  of  the  objects  of  Interest,  as  Indicated  in 
Chapter  1,  different  types  of  grammatical  rules  are  effective  for  de- 
scribing and  recognizing  different  objects.  For  example,  any  string- 
like object  (such  as  a road  or  a river)  is  recognized  more  efficiently 
by  a string  grammar  than  a tree  grammar.  Whereas  a tree  grammar  is 
necessary  for  recognizing  a complex  picture.  The  advantages  of  the  string 
grammar  approach  lie  In  its  systematic  grammatical  inference  procedure 
and  that  the  efficient  syntactic  analyzer,  as  finite  state  automata. 
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Figure  3.1.  Flowchart  for  an  image  recognition  system 
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There  are  two  major  methods  In  the  string  grammar  approach  for  Image 
recognition;  namely,  the  syntax-directed  method  and  syntax-controlled 
method. 

The  syntax-directed  method  involves  the  fol lowing  steps:  first,  an 
Inference  process  is  applied  to  a set  of  training  imagery  data  to  infer 
a set  of  grammatical  rules  which  in  turn  formalizes  a syntactic  model. 
Second,  based  on  this  model,  a set  of  template  matching  window  patterns, 
which  are  generated  by  this  grammar,  is  implemented  to  analyze  the  test 
images  and  to  recognize  the  object  of  interest.  Thus,  the  syntactic 
analyzer  is  a template  matching  process  which  is  directed  by  the  syntac- 
tic rules. 

The  syntax- control  led  method  is  comprised  of  the  following  steps: 
first,  a grammatical  inference  procedure  is  applied  to  a set  of  training 
imagery  data  to  infer  a set  of  grammatical  rules  which  In  turn  is  used 
to  construct  the  recognizer  or  automaton.  Then,  this  automaton  is  im- 
plemented to  analyze  the  test  images  and  to  recognize  the  object  of 
interest.  Hence,  the  syntactic  analyzer  is  an  automaton  which  is  direct- 
ly controlled  by  the  syntactic  rules.  1 

3.1.  SYNTAX-DIRECTED  METHOD  FOR  OBJECT  RECOGlilTlOH 

This  research  was  motivated  by  the  need  for  a method  which  can  fully 
automate  the  recognition  of  such  objects  as  highways,  rivers,  bridges, 
and  commercial/industrial  areas  from  satellite  images  such  as  those  of 
LANDSAT.  The  statistical  pattern  recognition  techniques  which  had  been 
developed  prior  to  this  work  had  not  shown  satisfactory  results.  For 
example,  the  land-use  classification  of  LANDSAT  images  has  been  studied 
by  Todd  and  Baumgardner  using  spectral  analysis  [84].  It  has  been  shown 
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that  highways  and  other  concrete  areas,  such  as  parking  lots,  could  not 
be  distinguished  from  each  other  due  to  the  fact  that  both  have  similar 
spectral  characteristics  in  spectral  analysis.  The  util ization  of  the 
syntactic  method  to  describe  spatial  relationship  among  different  ob- 
jects was  suggested  by  Fu  [33.  Some  research  was  done  on  LAHDSAT  images 
by  Brayer  and  Fu  [14].  They  were  able  to  program  a computer  to  analyze 
a city  scene  by  constructing  a hierarchical  graph  model  which  contains 
spatial  distributions  of  all  classes  in  the  scene.  Web  grammars  were 
used  to  describe  spatial  relationships  between  various  objects  in  the 
scene.  L?  and  Fu  [15]  started  with  pointwise  statistical  classification 
of  LANDSAT  images  and  then  appl ied  a tree  system  approach  to  the  LANDSAT 
data  interpretation.  Bajcsy  and  Tavakoli  [8l]  designed  a computer  pro- 
gram from  the  relational  graph  viewpoint  to  recognize  objects  from 
satellite  pictures.  The  research  undertaken  here  applies  to  the  recogni- 
tion of  certain  specific  objects  in  LANDSAT  images.  Th  ' syntax-directed 
method  utilizes  a set  of  finite-state  string  grammar  rules  to  describe 
and  recognize  the  objects  of  interest. 

3.1.1.  Grammatical  Inference  Process 

Based  on  knowledge  of  highway  structures,  several  initial  grammatical 
rules  were  written.  Then  a training  area  was  selected  (which  in  this 
case  was  Lafayette,  Indiana)  and  a preprocessed  training  image  was  ob- 
tained. The  initial  rules  generated  a set  of  pattern  windows.  The 
training  image  was  then  matched  to  the  set  of  generated  pattern  windows 
to  obtain  the  processed  result.  A highway  map  was  used  to  evaluate  the 
processed  result.  For  the  highway  structures  which  existed  in  the  map 
but  not  in  the  processed  result,  the  grammatical  rules  to  generate  those 
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matching  patterns  were  added  to  the  initial  set  of  rules  and  the  image 
was  then  reprocessed.  After  several  interactive  steps  the  final  set  of 
grammatical  rules  was  obtained.  The  primitives  for  the  grammar  are 
chosen  as  a,  b,  c,  d,  e,  f,  g,  and  h.  These  primitives  are  designed  as 
2x2  pixel  blocks. 


a + e i 

b it  f k 

c -*■  g «- 

d x h * 


For  example,  a window  for  an  intersection  of  highways  might  be 


This  syntactic  analyzer  was  the  window  operation,  which  processes  the 
image  window  by  window.  The  movement  of  the  window  is  to  shift  one 
column  or  one  row  at  a time.  Then  multibranch  patterns  can  be  represent- 
ed by  one-branch  grammar  rules.  For  example,  the  window  pattern  men- 
tioned above  can  be  analyzed  as  the  following  two  window  patterns: 


36 


The  one-branch  grammar  rules  are  as  follows: 

(1)  S+dA3  A3-k1A3  A^A^  A^  A^-vc 

" (2)  S-^cAj  Aj->cAj  Aj-*cA2  *2* cA2  A2*c 

These  rules  are  finite-state  grammar  rules  [2,  110]. 

A junction  of  two  highways  is  as  follows 


can  be  analyzed  by  two  window  patterns  that  are  generated  by  one-branch 
grammar  rules.  The  two  patterns  are. 


Then  the 

resultant  grammar  rules  can  be  expressed  in 

terms  of 

f inite- 

state  string  grammar  rules  as 

follow. 

The  grammar  G 

is 

G = (V  V 

P,  S) 

VN  « (S,  A,, 

^2»  Ag » Aij 

, Ag,  Ag,  Ay, 

A8»  Ag»  A 

*»> 

VT  » {a,  b,  c 

, d,  e,  f , 

9,  h} 

where  P: 

S->cAj 

A j ->c  A | A'j->cA2 

A2*>cA2 

A2^dA2 

A2^A2 

A2^A: 

S-»dA3 

A3-*dA3  A^A^ 

VfAi» 

Ai,^A, 

S-*eAj- 

Ar+eA_  AP-»eAr 

5 5 5 6 

V*a6 

VfA6 

A6'vgA6 

S-*fAy 

Ay*fAy  Ay->fAg 

Aj*fA8 

Ag-^gAs 

a8^a8 

V*A| 
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s+gA9 

VA10 

o 

<r 

o 

<~ 

VaAIO 

'VfA10 

VhA 

a2->c 

A2<-d 

Aj.e 

A2->-b 

A il+& 

A,  -ve 
'4 

Vf 

Vc 

Ve 

vf 

A6-g 

A6*d 

A6  +cA6 

Vc 

Aj.f 

V9 

Vh 

Ve 

A,0*9 

A10*a 

Al<ff- 

3.1.2.  Syntax-Di rected  Analys Is 

The  syntax-directed  method  consists  of  two  levels,  namely,  pre- 
processing and  syntax-directed  grammatical  analysis.  The  transformation 
processor  transforms  the  rnul ti spectral  LANDSAT  Images  into  a single  bi- 
nary image.  The  syntax-directed  analyzer  then  analyzes  the  transform'?:* 
Image  based  on  a set  of  template  matching  patterns.  Structures  wt  ;.,n 
are  matched  by  this  set  of  templates  are  accepted,  otherwise  they  are 
rejected.  The  details  of  the  preprocessing  and  syntax-directed  gram- 
matical analysis  are  given  below: 

(l)  Preprocessing 

A.  Thresholding  Process:  First  the  LANDSAT  images  were  defined  in 
Euclidean  n-dimensional  space  E°.  (The  number  n represents  the  number 
of  channels  to  be  chosen).  \ pixel  is  described  by  an  ordered  n-tuple 
(xj,  x^,  ...,  x^).  LANDSAT  measurements  from  channel  1 and  2 are  very 
sensitive  to  concrete  areas.  A training  area  was  chosen  to  establish 
the  thresholds  of  the  spectral  intensity  of  concrete  areas  In  channel 
1 and  2.  Then  a threshold,  H,  was  obtained  from  the  sum  of  two  thres- 
holds from  these  channels.  Since  channels  3 and  A are  infrared  bands 
[82],  measurements  from  these  two  channels  are  sensitive  to  thermal 
emitting  objects.  Watery  areas  are  strictly  non-thermal  emitting 
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objects.  The  negative  reaction  In  the  infrared  bands  causes  contrast 
and  makes  the  extraction  of  watery  areas  from  channels  3 and  h easier. 
The  same  procedure  of  threshold  finding  as  that  for  concrete  areas  is 
applied  here  to  obtain  threshold,  R,  for  river  recognition.  The  trans- 
formation process  works  in  such  a way  that  if  the  sum  of  the  spectral 
intensities  of  the  pixels  in  the  same  position  in  two  channels  is  great- 
er than  the  sum  of  the  two  thresholds  from  the  training  area  (for 
example,  channels  1,  2,  and  threshold  H for  highway  recognition,  and 
channels  3,  h,  and  threshold  R for  river  recognition),  the  pixel  is  set 
to  1;  otherwise,  it  is  set  to  0.  (For  river  recognition,  the  one-zero 
settings  are  reversed).  Thus,  the  mul tispectral  Images  are  transformed 
to  a single  binary  image. 

It  is  true  that  both  visible  bands  (channels  l and  2 are  sensitive 
to  the  concrete  spectra.  But  in  real  world  images,  the  influence  of 
neighboring  objects  sometimes  causes  the  deformation  of  the  intensity 
of  the  object  of  Interest  (such  as  a highway).  But  when  there  is  only 
one  channel  (image)  available,  the  thresholding  process  can  be  designed 
by  setting  the  threshold  on  that  one  image.  Experiments  of  this  kind 
were  also  conducted  on  other  objects  and  it  was  shown  that  by  using  the 
sum  of  the  spectral  intensities  of  two  visible  channels  (for  highways) 
one  obtains  a more  reliable  result  than  that  obtained  by  just  setting  a 
threshold  on  one  channel.  From  the  experiments  of  highway  recognition 
on  different  LANDSAT  images,  it  was  shown  that  some  highways  are  clear 
in  channel  1 and  some  in  channel  2.  A weighted  sum  of  threshold-finding 
could  not  be  achieved  but  the  results  of  summing  the  thresholds  from  the 
two  channels  were  satisfactory.  Hence,  the  sum  of  the  thresholds  from 


i 


| roil-.**-*— * 


39 


the  two  channels  was  used  as  the  threshold  process  In  the  experiments 
which  will  be  shown  here  and  in  section  3.1.3. 

' B.  Line  Smoothing  Process:  After  the  thresholding  process  is  com- 
pleted, a line  smoothing  technique  is  applied  to  remove  deformation  and 
re-establish  continuity  of  the  lines.  For  a given  center  pixel  of  a 
3x3  window,  the  operation  starts  from  the  left  upper  corner  pixel,  sf 
It  is  one,  the  column  is  shifted,  it  is  zero,  the  surrounding  eight 
pixels  are  checked,  if  there  exists  at  least  two  "l's"  which  are  not 
adjacent  to  each  other,  then  a "1M  is  set  into  the  center  position.  The 
operation  continues  until  reaching  the  rightmost  column  of  the  digitized 
image.  Then,  the  operation  is  shifted  one  row  down  and  starts  from  the 
left  most  column  with  the  same  process  until  the  last  row  of  the  digitized 
image  Is  reached. 

(2)  Syntax-Directed  Grammatical  Analysis 

Input:  The  transformed  binary  image  which  is  a Q.(i,j)  memory  array. 
Output:  The  syntax-directed  result. 

Algorithm: 

Step  1 : Set  G(M,N)  to  be  an  operation  window  (8  x 8 in  size). 

Step  2:  Load  that  part  of  the  array  0.(1, J)  in  which  J = 1,  8; 

I =1,  8 bn  the  operation  window  G(M,N). 

Step  3:  Compare  the  operation  window  with  the  set  of  template 

matching  window  patterns  (Figure  3*2)  which  were  gener- 
ated by  string  grammar,  G.  (Thirty  four  templates  were 
used  in  this  analysis.  The  templates  were  obtained  from 
the  inference  process.  In  the  inference  process,  new 
templates  are  added  to  the  set  of  templates  in  each  in- 
teractive process.  The  number  of  templates  used  here 
(34)  Is  the  number  of  templates  in  the  final  interactive 
process  of  the  inference  process.  From  the  experimental 


Figure  3.2.  A set  of  template  matching  8x3  window  patterns 
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result,  these  34  templates  proved  adequate  for  these 
applications.)  If  the  operation  window  belongs  to  any 
one  of  the  pattern  In  that  set  of  patterns,  the  prfmi- 
•'*  tlve  pattern  in  that  window  Is  accepted,  and  stored  in 

the  resulting  memory  array  R(|,J).  If  it  dees  not  be- 
long to  that  set  of  patterns,  then  go  to  step  4. 

Step  4:  Shift  one  column  to  the  right  of  Q(l,J)  in  step  3.  Then 
go  to  step  3 and  continue  until  reaching  the  rightmost 
column. 

Step  5:  Shift  one  row  downward;  go  to  step  3,  until  reaching  the 
last  row  of  the  digitized  image.  Syntactically  correct 
structures  are  recognized  and  stored  in  the  resultant 
memory  array  R(I,J). 

Step  6;  Output  the  result,  R(l,J). 

The  flow  chart  for  highway  recognition  by  the:  syntax-directed  method 
is  given  In  Appendix  B.  Since  rivers  have  the  same  linear  features  as 
highways,  the  inferred  grammar  for  highways  can  also  be  used  for  rivers. 
This  assumption  is  justified  by  the  results  of  the  experiments  on  river 
recognition.  The  flow  chart  for  river  recognition  is  also  provided  in 
Appendix  C. 

Spectrally  speaking,  bridges  and  commercial/industrial  areas  have 
similar  characteristics  to  concrete  parking  lots  and  highways.  This 
means  that  the  existing  statistical  techniques  are  inadequate  for  the 
distinction  of  bridges  from  highways  and  for  the  recognition  of  commer- 
cial/industrial areas  as  such.  The  syntactic  object  recognition  tech- 
nique uses  the  spatial  relationship  to  distinguish  highways  from  other 
concrete  areas  by  the  syntax-directed  method,  and  then  uses  semantic 
information  as  a postprocessor  to  distinguish  birdges  and  commercial/ 
Industrial  areas  from  the  recognized  highways.  For  bridge  recognition. 
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first  the  images  are  processed  by  the  syntax-directed  method  for  highways 
and  rivers.  Then  a logical  "AND"  operation  Is  applied  to  the  highway 
results  "H"  and  the  river  result  "R"  to  obtain  the  bridge  recognition  re- 
sult "D".  This  operation  results  In  a "bridge"  which  is  the  Intersection 
of  a "highway"  and  a "river".  A flowchart  of  bridge  recognition  is' 
given  in  Appendix  D.  The  length  of  the  recognized  bridge  can  be  calcu- 
lated by  the  following  algorithm: 

Algorithm  for  calculation  of  bridge  length  and  coordinates,  (from  LANDSAT 
data) 

Input:  Recognized  bridge  result. 

Output:  The  calculated  length  and  coordinates  of  the  bridge. 

Algorithm: 

(I)  Calculate  the  number  of  horizontal  rows  which  have  at  least 
one  brtuge  pixel.  The  value  is  a. 

(if)  Calculate  the  number  of  vertical  column  v/hfch  have  at  least 
one  bridge  pixel.  The  value  is  b. 

(iii)  if  a » 1,  the  length  of  the  bridge,  c,  is  b x 56  meters. 

If  b = ?,  the  length  of  the  bridge,  c,  is  a x 79  meters. 
Otherwise  go  to  (iv). 

(iv)  The  length  of  the  bridge,  c,  « \J(a  x 79) 2 + (b  x 56) 2. 

The  idea  of  this  algorithm  Is  to  calculate  the  hypotenuse  of  a right  tri- 
angle, using  the  information  that  each  pixel  In  a LANDSAT  image  corres- 
ponding to  79  meters  in  vertical  length  and  56  meters  in  horizontal  length. 
Step  (rii)  is  the  case  in  which  a bridge  Is  right  on  the  horizontal  row 
or  vertical  column.  The  coordinates  of  the  recognized  bridge  can  also 
he  determined  from  the  recognized  bridge  pixels,  as  the  coordinates  of 
the  pixels  in  the  Image  can  be  directly  related  to  the  global  coordinates 
of  meridian  anf  elevation. ■ . 
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For  commercial/industrial  area  recognition,  first  the  images  are 
processed  to  obtain  highways.  The  highway  recognition  result  is  then 
deleted  from  all  concrete  areas  by  the  Boolean  "Exclusive  OR"  operation. 
Then  for  the  four  by  four  cells  of  the  image  after  the  "Exclusive  OR" 
operation,  if  over  60%  of  the  pixels  in  the  cell  are  concrete,  this  cell 
area  is  called  commercial/industrial.  This  is  because  the  commercial/ 
industrial  area  has  a high  density  of  concrete  buildings  and  concrete 
storage  yard.  The  size  of  the  recognized  commercial/industrial  area 
and  the  coordinates  of  the  center  of  this  area  can  be  calculated  by  the 
following  algorithm: 

Algorithm  for  calculation  of  the  area  of  a commercial/industrial  area 

and  its  center,  (from  LAUDSAT  data) 

input:  Commercial/industrial  recognition  result. 

Output:  The  calculated  area  and  the  center  of  the  commercial/ 
industrial  area. 

Algorithm: 

(?)  Calculate  the  number  of  commercial/industrial  area  pixels. 
The  value  is  n. 

(ii)  The  area  equals  n * (0.079  x 0.056)  square  kilometers. 

(iii)  For  every  commercial/industrial  area  pixel  C.(i,J)  calculate 
n n * 

( Z l.)/n.  The  value  is  P.  Calculate  ( Z J.)/n.  The 
i=l  i=l 

value  is  q. 

(iv)  The  center  of  the  commercial/industrial  area  Is  the  coordi- 
nate (P,q). 

The  resolution  of  LANDSAT  images  is  79  x 56  meters  per  pixel.  Hence,  the 
area  of  each  pixel  is  (0.079  x 0.056)  square  kilometers.  The  center  cf 
cowiercial/industrial  area  is  the  coordinate  which  has  the  x coordinate 
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as  the  mean  of  all  the  x coordinates  of  the  commercial /Indus trial  pixels 
and  the  y coordinate  as  the  mean  of  all  the  y coordinates  of  the  commer- 
cfal/industrial  area  pixels. 

3.1.3.  Computer  Experimental  Results  on  Recognition  of  Highway  and  River 
from  Satellite  Images 

The  syntax-directed  method  was  implemented  by  FORTRAN  programming 
on  the  IBM  360/67  computer  at  the  Laboratory  for  Applications  of  Remote 
Sensing  (LARS)  at  Purdue  University. 

(l)  Highway  Recognition 

Figure  3.3(a)  is  a LANDSAT  image  of  the  Indianapolis,  Indiana  area. 
The  platform  altitude  at  which  the  picture  was  taken  is  306,200  feet  and 
there  are  128  grey  levels  used  In  the  digitized  image  [82].  Figure  3.3 
(b)  Is  the  intermediate  output  after  line  smoothing  in  the  transformation 
process.  The  highway  recognition  result,  by  the  syntax-directed  method. 
Is  shown  in  Figure  3.3(c).  This  area  is  a 96  x 96  pixel  Image  which 
shows  the  junction  of  Interstate  highway  65  (northwest  to  southeast)  and 
highway  46$  (north  to  south)  in  the  left  upper  part  of  Figure  3.3(d). 

The  exact  location  of  Figure  3.3(a)  is  labeled  on  Figure  3.3(d)  as  the 
square  outlined.  The  experimental  results  indicate  that  the  syntax- 
directed  method  Is  rather  successful. 

Figure  3.4(a)  is  a LANDSAT  image  of  the  downtown  area  of  Chicago, 
Illinois.  Figure  3.4(b)  is  the  highway  recognition  result  from  Figure 
3.4(a)  by  the  syntax-directed  method.  This  area  Is  the  junction  of  the 
Chicago  Skyway  and  the  Dan  Ryan  Expressway  (highway  94)  which  Is  shown 
in  the  downtown  map  of  Chicago  Figure  3.4(c).  Lake  Shore  Drive  is 
also  recognized,  but  as  the  resolution  of  the  LANDSAT  satellite  sensor 
Is  only  79  x 56  meters  per  pixel,  the  recognition  Is  not  perfect. 


because  whenever  the  widths  of  highways  ere  less  than  the  resolution 
values,  the  recognition  will  be  unfavorably  affected.  Figure  3.5(a)  is 
the  LAHDSAT  image  of  Harvey,  Illinois.  The  computer  result  of  highway 
recognition  by  the  syntax-directed  method  is  shown  in  Figure  3.5(b). 

For  comparison  of  the  accuracy  of  the  result,  the  city  map  of  Harvey, 
Illinois  is  shown  In  Figure  3.5(c).  The  recognized  highways  are  high- 
way 80  (southwest  to  northeast),  highway  57  (south  to  north)  and  state 
road  50  (south  to  north).  The  ,,H"  in  Figures  3.3(c),  3. Mb)  , and  3.5(b) 
stands  for  a highway  pixel. 

(2)  River  Recognition 

For  the  purpose  of  showing  that  this  method  works  also  for  river 
recognition,  a terrain  area  northeast  of  San  Francisco,  California  was 
processed  by  the  syntax-directed  method  for  river  recognition.  The 
LAHDSAT  image  is  shown  in  Figure  3.6(a).  The  river  recognition  result 
given  in  Figure  3.6(b),  shows  that  the  syntax-directed  method  sucessfuily 
recognized  a winding  river  in  that  image.  The  size  of  this  image  fs 
also  96  x 96  pixels.  The  topographic  map  for  the  area  is  shown  in  Figure 
3.6(c) . 

Figure  3.7(a)  Is  a LANDSAT  image  of  the  Lafayette,  Indiana  area. 

The  river  recognition  result  is  shown  In  Figure  3.7(b).  The  "R"  in 
Figure  3.7(b)  stands  for  a river  pixel.  Figure  3.7(c)  is  a city  map  of 
Lafayette.  The  river  recognition  result  shows  the  Wabash  river  through 
the  Lafayette  area.  The  Wabash  river  divides  the  area  into  West 
Lafayette  and  Lafayette.  The  1 ighter  areas  in  the  LAHDSAT  image  are  the 
concrete  areas  and  the  darker  areas  are  the  watery  areas.  The  bright 
line  from  north  to  south  in  Figure  3.7(a)  is  highway  65,  and  the  dark 
line  from  northeast  to  southwest  Is  the  Wabash  River. 
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Figure  3.3(e).  Satellite  Image  of  northwest  part  of 
Indianapolis,  Indiana. 
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Figure  3.3(b).  Intermediate  output  after  line 

smoothing  process  on  Figure  3.3(a) 
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Figure  3.3(c)*  Highway  recognition  result  from  Figure  3*3(a) 
by  syntax-directed  method. 
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Figure  3.3(d),  City  map  of  Indtanapol Is,  Indiana 
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Figure  3.4(b). 


Highway  recognition  result  from  Figure  3.4(a) 
by  syntax-directed  method. 
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Figure  3. Me).  Map  of  downtown  Chicago,  Illinois. 
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Figure  3.5(a).  Satellite  Image  of  Harvey,  Illinois. 
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Highway  recognition  result  fron  Figure  3.5(a)  by 
syntax-direct  i method. 
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Fig.  3.5(a) 


Figure  3.5(c).  City  map  of  Harvey,  Illinois. 


Figure  3.6(a).  Satellite  Image  of  north  part  of 

San  Francisco  Bay  area,  California. 
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Figure  3.6(b)* 


River  recognition  result  from  Figure  3.6(a) 
by  syntax-directed  method. 
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Figure  3.7(a).  Satellite  Image  of  Lafayette  area, 
(192  x 192  pixels). 
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Figure  3.7(b).  River  recognition  result  from  Figure  3.7(a)  by 
syntax-directed  method. 
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Figure  3.7(c).  City  nap  of  Lafayette,  Indiana. 
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3.1.4.  Computer  Experimental  Resul ts  on  Recognition  of  Bridges  and  Com- 
mercial/Industrial Area  from  Satellite  Images 

(1)  Bridge  Recognition 

Figure  3.8(a)  is  a topographic  map  of  the  lower  part  of  Figure  3.3 
(a);  Indianapolis,  Indiana.  It  shows  in  the  left  part  of  the  map,  that 
there  is  a bridge  over  Eagle  Creek  Reservoir.  The  bridge  recognition 
result  by  the  syntactic  object  recognition  method  given  in  Figure  3.8(b) 
shows  that  the  bridge  was  successfully  recognized  and  that  the  length  was 
calculated  to  be  672  meters.  In  the  left  lower  part  of  Figure  3.8(a)  the 
scale  of  the  map  is  provided.  The  length  shown  in  the  nap  is  about  the 
same  as  that  found  by  the  syntactic  method.  The  coordinates  of  the 
bridge  can  also  be  located  by  this  method. 

Another  experiment  was  conducted  in  the  Lafayette  area  pertaining 
to  bridge  recognition.  Referring  back  to  Figure  3.7(c),  it  can  be  seen 
that  there  is  a bridge  on  Highway  1-65  over  the  Wabash  River.  The 
LANDSAT  image  shown  in  Figure  3.9(a)  processed  by  the  syntax-directed 
method  for  bridge  recognition  and  the  result  is  shown  in  Figure  3.9(b). 
The  recognized  bridge  is  in  the  right  lower  part  of  the  figure.  Its 
length  was  calculated  to  be  454.1  meters.  The  coordinates  of  the  loca- 
tion of  the  bridge  are  also  given  in  Figure  3.9(b). 

(2)  Commercial/I ndustrial  Area  Recognition 

Experiments  were  conducted  in  several  different  areas.  Figure  3.10 
(a)  is  a LANDSAT  image  (9&  x 96  pixels)  taken  on  September  30,  1972  of  a 
section  within  the  northwestern  part  of  the  Indianapolis  area,  which  is 
a little  south  of  the  image  in  Figure  3.3(a).  Figure  3.10(b)  is  the 


topographic  map  of  the  area.  This  map  was  made  by  the  Department  of 


Natural  Resources  of  the  state  of  Indiana  in  1967.  In  Figure  3.10(c)  the 
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intermediate  output  after  preprocessing  Is  given.  Figure  3.10(d)  is  the 
highway  recognition  result.  The  intermediate  output  after  the  "Exclusive 
OR"  operation  in  postprocessing  is  given  in  Figure  3.10(e).  Figure  3.10 
(f)  is  the  commercial/industrial  area  recognition  result.  This  area  is’ 
identified  as  the  Lafayette  Square  Shop,  m-  Center  in  Indianapolis.  Figure 
3.10(g)  is  the  urban  development  information  extraction  result.  H,  R, 
and  C in  Figure  3.10(g)  represent  highway,  river,  and  commercial/ 
industrial  pixels,  respectively.  Interstate  highway  65  (upper  right  part) 
goes  into  the  city  from  the  northwest.  Highway  465  (from  north  to  south) 
surrounds  the  city  and  highway  74  (left  lower  part)  goes  into  the  city 
from  the  west.  The  Lafayette  Square  Shopping  Center  is  located  in  the 
suburb  and  close  to  the  highways.  The  commercial /Indus trial  area  is 
automatically  calculated  to  be  57-75  square  kilometers.  The  center  of 
the  commercial/industrial  area  is  calculated  to  be  in  coordinate  (66,63) 
of  the  96  x 96  image  frame.  This  information  was  also  automatically  ex- 
tracted by  the  method. 

Comparing  Figure  3.10(g)  with  the  topographic  map  Figure  3.10(b)  of 
1967,  we  can  observe  the  growth  of  the  commercial  area  clearly  on  the 
northern  and  eastern  parts  of  the  shopping  center.  This  indicates  that 
the  proposed  method  could  be  useful  in  topographic  map  making  and  up- 
dating. 

Several  large  images  (192  x 192  pixels)  were  also  processed  by  the 
same  technique.  Figure  3.11(a)  is  the  commercial/industrial  area  recog- 
nition result  from  the  image  of  Lafayette,  Indiana  (compare  Figure  3.7 


i 


} 


(a)).  Figure  3.11(b)  is  the  urban  development  information  extraction 
result.  The  recognized  commercial/industrial  areas  In  Figure  3.11(a)  are 
marked  on  the  Lafayette  map  Figure  3.7(c).  The  comnerciaf/industria! 


Figure  3.8(a).  Topographic  map  of  lower  part  of  Figure  3.3(a) 
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Figure  3.8(b).  Bridge  recognition  resuit  from  Figure  3.3(a) 
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Figure  3.9(b).  Bridge  recognition  result  of  Figure  3.9(«). 


Fiqure  3.10(a).  Satellite  image  of  northwest  part  of  Indianapolis 
Indiana  area  (96  x 96  pixels). 
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Figure  3.10(c). 


Intermediate  output  after  preprocess  of 
Figure  3.10(a). 
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Figure  3.10(d).  Highway  recognition  result  on  Figure  3.10(a). 
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Figure  3.10(e).  i.^errr.edlate  output  after  exclusive  or  operation 
(EXOl.)  of  the  postprocessor. 
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Figure  3.10(f).  Commereial/industrlal  area  recognition  result. 
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Figure  3. 10(g) . 


Urban  development  Information  extraction  result 
of  Figure  3.10(a). 
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Figure  3.11(a).  Commercial /industrial  area  recognition 
result  from  Figure  3.7(a). 
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.11(b).  Urban  development  Information  extraction 
result  of  Figure  3.7(a). 
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areas  of  the  Anheuser-Busch  corn  refining  plant*  the  National  Homes  Com- 
pany, the  Alcoa  Aluminum  Plant,  the  West  Lafayette  Village  and  Levee 
shopping  areas,  the  Woolco  Shopping  Center,  and  the  Lafayette  downtown 
area  are  successfully  recognized. 

3.2.  SYNTAX-CONTROLLED  METHOD  FOR  OBJECT  RECOGNITION  , 

The  syntax-controlled  method  is  a method  which  utilizes  an  automaton 
(or  a parser)  as  a recognizer  (or  a syntax  analyzer)  to  recognize  the 
patterns  of  interest.  The  syntax-controlled  method  for  image  recognition 
differs  from  the  syntax-directed  method  in  the  grammatical  inference  pro- 
cedure and  the  syntactic  analysis.  The  grammatical  inference  procedure 
for  the  syntax-controlled  method  is  a fully  computer  automated  finite- 
state  grammar  inference  procedure  whereas  for  the  syntax-direct*  method, 
the  grammatical  inference  procedure  is  an  interactive  process.  » r.  _ /n- 
tactic  analyzer  for  the  syntax-controlled  method  is  a determini- 
finite-state  automaton  v/hich  will  be  described  in  section  3.2.2. , whereas 
for  the  syntax-directed  method,  the  syntactic  analys  is  a template- 
matching process. 

3.2.1.  Grammatical  Inference  Procedure 

In  order  to  describe  a class  of  patterns  precisely,  grammatical 
rules  are  inferred  from  a set  of  sam,  le  patterns.  A k-tail  method  [20] 
which  will  be  described  in  this  section  was  implemented  in  this  study. 

The  grammatical  rules  which  describe  a class  of  patterns  are  automatical- 
ly inferred  from  a set  of  training  sample  patterns  by  the  computer  pro- 
gram of  the  k-tail  method.  Before  describing  this  technique,  some 
definitions  need  to  be  given  [20,110]. 
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Definition  1.  The  formal  derivative  of  a set  of  strings.  A,  with 
respect  to  the  symbol  a £ V-j.  is  defined  as  DaA  = 
{x|ax  € A}. 

Definition  2.  If  a]f  a2,  a^j  an  is  a string  then,  Da^, 

•**»  Vlan  A = Dan^Dai»  a2*  ***»  Vl  A** 

JU 

Definition  3.  Let  u = a^,  a2>  •••»  a,_  •€  V^.  and  let  A C L(G). 

The  k-tail  of  A with  respect  to  u is  defined 
as  (u,  A,  K)  = {x|x6  DuA,  [xj  < K}  where  jxj 
is  the  length  of  the  string. 


Definition  h. 


Let  u.  and  u.  be  two  distinct  states  of  the 
i J 

canonical  derivative  finite-state  grammar  G p. 

These  two  states  are  associated  with  the 

derivatives  Dx.S*"  and  Dx.S*"  respectively  where 
1 J * 

x.  and  Xj  are  sequences  from  V.p  The  two 

states  u.  and  u.  are  said  to  be  k-tail 

1 J / t \ 

equivalent  if,  and  only  if,  g(x.,S  ,K)  = 

g(x.,st,K). 


Definition  5.  A non-deterministlc  finite-state  automaton  is  a 

5-tuple  H = (Q,  E,  6,  qQ,  F),  where 

(1)  Q is  a finite  set  of  states; 

(2)  E is  a finite  set  of  permissible  input  symbols; 

(3)  is  a mapping  function  from  (I  x E to  p(Q) 

which  dictates  the  behavior  of  the  finite- 
state  control,  (6  is  sometimes  called  the 
state  trasition  function); 

(k)  q in  Q is  the  initial  state  of  the  finite- 
state  control;  and 

(5)  F C is  the  set  of  fir  states. 


The  inference  procedure  for  the  grammar  which  describes  the  highway 
and  river  structures  is  a k-tail  method.  The  algorithm  for  this  method 


is  as  follows: 
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Algorithm  of  k-tail  grammatical  inference  procedure  for  finite-state  string 


grammar. 

input:  Training  sample  patterns. 

Output:  Finite-state  string  grammatical  rules. 

Algori thm: 

(i)  input  training  sample  patterns  and  encode  as  the  string  of 
primitives  a,  b,  d,  e,  f,  g,  and  h. 


(ii)  Find  first  order  derivatives  of  the  set  of  all  strings  A 
with  respe-t  to  the  primitive  set  a,  b,  c,  d,  e,  f,  g,  and 
h.  (e.g.,  •»..  3 D_A  ® (xjax  € A},  u^  = D_A,  u2  = D^A,  - 

DcA’  Ui»  = DdM>  u5  = °eA’  u6  = DfA»  U7  = DgA*  at'd  u8  = V* 

(iii)  Find  second  order  derivatives  of  the  set  of  ail  strings  A 
with  respect  to  the  primitive  set.  (e.g.,  u„  = D A = D 

jJ  33  3 

(D  A)  - {x|aa,c  •£  A}. 

3 

U9  = DaaA’  U10  " °abA*  U11  = DaCA’ Ul6  = DahA* 

U17  = DbaA>  Ul8  = DbbA’  U19  “ DbcA»  ***’  = °bhA» 

u25  = DcaA’  U26  = DcbA*  U27  = °ccA °32  = DchA» 

U33  = °daA’  U3;*  = DdbA’  U35  = DdcA*  ****  U*»0  = V* 

Uh\  = DeaA’  V “ DebA’  Ui»3  = DecA’  ***»  °*s8  “ DehA’ 

uij9  = DfaA’  U50  = DfbA’  U51  = °fcA*  ***’  U56  = DfhA» 

u57  = %A’  U53  = DgbA>  u59  = DgcA’  ***’  U64  " DghA’ 

u65  = DhaA’  u66  = DhbA’  u67  = DhcA’  ***»  u72  = DhhA* 

(iv)  Find  third  order  derivatives  of  the  set  of  all  strings  A, 
with  respect  to  the  primitive  set  a,  b,  c,  d,  *».  f,  g,  and 
h.  (e.g.,  u?i(  = l)aabA  = °b (°a <DaA) ) = (x|aabx  -6  A}). 

u73  ’ Daaafl-  > " DaabA “80  * °,ahA’ 


<30 


u8l  “ DabaA’  U82  c DabbA u58  " DabhA’ 


u568  = dhgaA*  “563  ~ u 


hgb 

3hhb; 


575  ~ °hghA» 


u576  " DhhaA’  u577  “ DhhbA»*-*»  °584  = °hhhA* 


(v)  Find  nth  order  derivatives  of  the  set  of  all  strings  A, 

with  respect  to  the  primitive  set  (N  is  the  maximum  length 
of  string) . 

(vi)  Find  canonical  derivative  finite-state  grammar. 

GCD  = ^VT’  VN*  R’  ° ^ 

\l  is  the  primitive  set,  \lj  = {a,  b,  c,  d,  e,  f,  g,  hi 

VN  = {ur  V u3’  ***’  u584} 

o ~ {Uj,  u^,  ...»  Ug} 

The  rule,  R,  is  defined  as  follows:  let  u.,  u.  ■€  V., 

1 j N 

u.  ->  a u.  if,  and  only  if,  D u.  = u. 

1 j ’ ' a 1 j 

u.  -»•  a if,  and  only  If,  X 6 D u. 
i ' a 1 

(e.g.,  D u.  = uQ  produces  the  grammatical  rule  ut  a u , 
a 1 j I y 

and  Dbuj0  ~ ug2  Prot'uces  t*',e  grammatical  rule  UjQ  -»■  bUg^) . 

(vii)  Find  equivalence  classes  of  states.  Find  the  length  of  all 

the  u.  where  i = 1 to  584. 

1 

(viii)  Find  1 — ta ? 1 equivalence  classes.  If  ju.  J >1  and  |u.|  > 1, 

then  u.  - u.. 
i J 

(ix)  Find  2-tail  equivalence  classes.  If  |o.  | >2  and  ju.|  > 2, 

. ,en  u.  - u..  If  I u , { < 2,  then  u.  has  no  equivalence 
1 j * i 1 “ 1 

classes. 

(x)  Find  (n-l)-tail  equivalence  classes.  If  |u.  | > (ii-1)  and 
|u.|  > (H-l),  then  u.  = u . If  ju.|  < (M-l)  then  has  no 
equivalence  classes. 
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(xi ) Find  the  derived  grammar,  GQ  = (V{J,  V^,  P,  S) , from  the 

canonical  derivative  grammar,  G^,  and  the  minimized  state 
state  set,  Uj,  by  the  equivalence  classes. 

a.  The  terminal  set,  V^,  is  the  same  for  G^  and  G^ . 

b.  The  non-terminal  set,  V^,  corresponds  to  the  distinct 
blocks  of  a partition  of  V^c  by  the  equivalence  classes. 

c.  S is  the  starting  symbol  ond  corresponds  to  the  block 
which  contains  c. 

d.  A grammatical  rule  of  P is  B.  -*■  atK  if,  and  only  if,  there 

exists  u.,  u.  € V such  that  u.  -*•  a u.,  u.  € B.,  u.  € B.. 

i J M t j * i iJJ 

e.  A grammatical  rule  of  P is  B.  ■+  a if,  and  only  tf,  there 

exists  u.  € V..  such  that  u,  ->•  a and  u.  € B_. 

i N { i i 

(xii)  Find  the  derived  grammars,  GQ,  based  on  the  partitions  of 
1-tail  equivalence,  2-tail  equivalence,  and  (N-l)  tail 
equivalence. 

(xii I)  The  obtained  derived  grammars,  GQ,  are  the  resultant  finite- 
state  string  grammatical  rules  corresponding  to  the  different 
values  of  k in  the  k-tail  method. 

This  algorithm  was  implemented  and  applied  to  Infer  the  highway 
grammar.  Sample  patterns  are  shown  in  Appendix  E.  The  canonical  deri- 
vative finite-state  grammatical  rules  are  inferred  as  the  intermediate 
output  of  the  f ini te-strate  string  grammar.  The  canonical  derivative 
grammatical  rules  are  as  follows. 


uy+c 

U27 

V*  u27 

Y°  U27 

V*  “28 

i 

u^c 

U27 

V0  "36 

Vc  u35 

V1  “36 

Vd 

U36 

u35 

Yd  U36 

V1  “36 

u^e 

u45 

Vf  Y 

Vs  u55 

VF  u54 

x 

i 

| 

f 


02 


Vf 

U5A 

V*  “55 

Vf 

u54 

Vf  u54 

U63 

V9  “63 

u^->f 

U62 

“t*9  “63 

U27‘>C 

u219 

un?c 

u27"° 

U219 

U219VC 

V* 

U220 

“220^ 

“27^ 

U22G 

W* 

u4,  <+ d 

c.  J 

U228 

u'2lfd 

U35"C 

U283 

U283^C 

U292 

“292’“ 

u36^c 

U29l 

u ~*~c 

291 

u]Cd 

U292 

“292-** 

u36"c 

U291 

“291""° 

V* 

U293 

U293‘>e 

u45^ 

U365 

U365‘>e 

u438 

u430"** 

u54^ 

U433 

“439^ 

u5  rf 

u438 

u438""f 

u54^ 

U439 

u439^ 

U5^e 

U437 

ui»37^ 

U55^ 

u447 

u44?">9 

u55->g 

u447 

uW7*fl 

u62*f 

u502 

U502^f 

u63^ 

U51i 

“sii** 

u63"f 

U510 

U5I0^f 

u63+f 

u510 

u510"^f 

U35"C 

U283 

“203'^ 

After  finding  the  equivalence  classes  of  states  Uj,  the  resultant 
finite-state  string  grammar  for  highv/ays  is  inferred  automatically  by 
computer.  The  resultant  grammar  is  ns  follows  (when  It  = 1 in  the  k-tail 
method) : 

Gk=i  = V P’  ^ 

Vf(  - (S,  A,  B,  C,  D,  E} 

VT  ^ (a,  b,  c,  d,  e,  f,  g,  h} 

P:  S->cS  S-hIS  S->eS 

S-+fS  S-+gS  S-*cA 
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t 

f 

\ 

I 

% 

| 

j 

| 

i 

5 


S-wdA 

S-*dB 

S-heC 

S-*fD 

S->gD 

S+fE 

S-K|E 

A-*c 

A->d 

B-xi 

C->e 

D->f 

D->g 

E-*f 

E+g 

The  finite-state  string  grammar  for  highways  when  k=2  in  the  k-tail 

method  is  also 

inferred  as  follov/s: 

Gk=2 

= (VN 

, vT,  P, 

s) 

VN 

= (s, 

A,  B,  C, 

D,  E,  F,  G,  H,  1,  J,  K,  L,  H,  N> 

VT 

" *a> 

b , c , d , 

e,  f,  g,  h) 

P: 

S->cA 

S->dB 

S->dD 

S-vcC 

S->eE 

S->fF 

S->gG 

S->gl 

S+fH 

A-+cJ 

A-»dJ 

frxIK 

C->cJ 

D-k1J 

IHcJ 

D-»eL 

E->eL 

F-*fM 

F->gH 

F->-eL 

G-*gM 

H->fN 

1-9-gN 

l**-fM 

J-*c 

J-»d 

K-*d 

L->e 

M+f 

H-hj 

IH-f 

IHg 

An  excellent  aspect  of  the  k-tail  method  lies  in  the  fact  that  the 

value  of  k can 

vary  depending 

on  the  case.  The  grammar  inferred  by  k=2 

is  more  precise  than 

the  grammar  inferred  by  k=l  in  describing  the  high- 

way  structures 

. However,  the 

computer  processing  time  of  grammatical 

inference  for 

k=2  i S 

longer  than  the  time  of  grammatical  inference  for 

k=l . The  CPU  time  for  grammatical  inference  of  k=l  Is  9 seconds.  The 
CPU  time  for  grammatical  inference  of  k=2  is  15  seconds  on  the  IBM  360/ 


l 
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67  computer.  The  grammatical  inference  procedure  of  k=3  v/as  also  imple- 
mented. The  CPU  time  is  270%  of  that  for  k=l . Thus,  the  case  of  k=3  is 
not  suggested  for  highway  grammar  inference.  Concerning  the  choice  between 
k=1  and  k=2  for  the  highway  grammar  inference,  several  computer  experi- 
ments of  highway  recognition  by  these  two  sets  of  gramnars  were  conducted. 
The  experiments  and  the  suggested  choice  of  k values  will  be  presented 
in  the  next  section. 

3.2.2.  Syntax-Controlled  Analysis 

The  syntax-controlled  method  consists  of  two  levels,  namely,  a 
transformation  process  and  the  syntax-controlled  analysis.  The  trans- 
formation process  of  the  syntax-controlled  method  is  the  same  as  that 
for  the  syntax-directed  method  given  in  section  3.1.2.  The  syntax- 
controlled  analysis  uses  a deterministic  finite-state  automaton  as  a 
recognizer  to  analyze  the  transformed  image.  Instead  of  comparing  the 
operation  window  G(M,M)  with  the  set  of  template  windows  as  in  section 

3.1.2,  the  window  G(M,N)  is  analyzed  by  the  finite-state  automaton.  The 
algorithm  is  as  follows: 

Algorithm  of  highway  recognition  by  syntax-controlled  method. 

Input:  LANDSAT  images. 

Output:  Highway  recognition  result  by  syntax-controlled  method. 

Algori thm: 

(i)  Transformation  Process:  same  as  for  syntax-directed  method 
as  in  section  3.1.2. 

(ii)  Fini te-state  Automaton  Analysis:  computer  automatically 
constructs  the  finite-state  automaton  from  the  Inferred 
grammar. 

(iii)  Construct  the  finite-state  automaton  M = {£,  Q,  6,  F). 

from  the  finite  state  grammar  G = (V^,  Vy,  P,  S)  as  follows  [2]: 
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a.  E = VT 

b.  Q = VN  U {T} 

c-  qD  * s 

d.  If  P contains  the  production  S-j-X,  then  F = {S,T}. 

Otherwise  F - {T } * 

e.  If  E->-a,  B -6  Vf],  a € V is  in  P of  grannar  G,  then 
6(B,a)  = T. 

f.  If  B-'-aC  is  in  P of  grammar  G and  C ■€  V^,  a € V^.,  then 

S(B,a)  - C and  6(T,a)  - 0 for  each  a € V^.. 

(iv)  Construct  the  deterministic  finite  automaton  M"  from  the 
finite  state  automaton  M as  follows  [2]: 

M"  = (Z't  r,  S',  qQ',  F')  (K  = (E,  Q,  5,  qQ,  F)) 

a.  The  states  (elements  of  Q^*)  of  M"*  are  all  the  subsets  of 
0.. 

b.  I'  = E 

c.  F'  i s the  set  of  all  states  in  <l'  containing  a state  of  F. 

d.  qQ'  - [qQ] , a state  of  M"  is  denoted  by  [q^,  q2»  ...t  q.] 

€ where  q^ , qj,  ...»  qj  € Q. 

e.  6'diqj,  ...,  qj},a)  « [Pj , ?2>  ...,  Pj)  if,  and  only  if, 

$([qj»  •••>  qj]  .a)  = ~ ^|»  ***» 

(v)  Set  G(H,ll)  to  be  an  operation  window  (8  x 8)  and  load  the 

array  of  the  transformed  image  from  Step  (l).  G(M,N)  equals 
0 ( I , J)  v/here  J = 1,  8;  and  1 = 1,8  for  the  first  case. 

(vi)  Encode  the  patterns  of  the  G(K,N)  window  into  the  string  of 
the  primitives  at,  b-%  c-*-,  dv,  e+,  f g-*-,  and  h*'  . 

(vii)  Send  the  string  to  the  deterministic  finite-state  automaton 
M".  If  the  automaton  accepts  the  string,  then  this  pattern 
is  accepted.  Therefore,  proceed  to  the  next  step,  otherwise 
this  pattern  is  rejected.  Proceed  to  the  next  step. 
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(viii)  Shift  one  column  to  the  right  of  0.(1, J)  In  step  (iii).  Then 
gc  to  step  (v)  and  continue  until  reaching  the  rightmost 
column. 

(ix)  Shift  one  row  downward  in  step  (v);  go  to  step  (v)  until 

reaching  the  last  row  of  the  digitized  image.  Syntactically 
correct  structures  are  recognized  and  stored  in  the  resultant 
memory  array  R(l,J). 

(x)  Output  the  result,  R(1,J),  which  ?s  the  result  of  highway 
recognition  by  the  syntax-controlled  method. 

For  the  highway  grammar  (k~l ) , the  finite-state  automaton  is  auto- 
matically constructed  by  computer  as  follows: 

Finite-state  automaton  for  highway  recognition  Mj  = (E»  Q,  5,  qQ,  F) 

I “ (a,  b,  c,  d,  e,  f,  g,  h> 
a - vN  U (T> 

% " S 

F » {T} 

6:  6 (S,c)  = S 

6 (S,d)  = S 
6(S,e)  » S 
6(S,f)  = S 
«(S,g)  = S 
6(S,c)  = A 
5(S,d)  = A 
6(S,d)  = B 
6 (S,e)  = C 
S(S,f)  » D 
6(S,g)  = D 
6(S,f)  * E 


<5(A,c)  = T 

. «(B,d)  = T 

6 (A,d)  = T 
6(C,e)  = T 
<S(D,f)  = T 
6 (D,g)  = T 
6(E,f)  = T 
6(E,g)  = T 
S(T,a)  = $ 

6(T,b)  = <j, 

6(T,c)  = <J> 

5(T,d)  = $ 

5(T,e)  = $ 

6(T,f)  = $ 

6(T,g)  - 4. 

«(T,h)  = <j> 

The  corresponding  deterministic  finite-state  automaton  for  the  case 
of  the  grammar,  k=l , is  as  follows:  Mj'  = (Z'r  O',  o',  qQ',  F') 

(H,  = (E,  0,  6,  qo,  F)). 

Z — Z { a , b , c , d,  e,  f,  g,  h 3 

Q'  = VH  U {T , 4»)U  {[S,A],[S,A,B],tS,C],[S,D,E],[S,A,T],[S,A,B,T], 
[S,C,T],  [S,D,E,T] } 

V = 153 

F'  = {T, [S,A,T],  [S,A,B,f],  [S,C,T],  [S,D,E,T)} 

6([S,c],g)  = [S,D,E3 
«([S,D,E3,c)  = [S.A3 


5'  : <5 ( [S] ,c)  « IS, A] 
6([S3,d)  - [S,A,B] 
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6([S],e)  = [S,C] 

5([S,D,E],d)  = tS,A,B] 

6([S],f)  = [S,D,E] 

5([S,D,E],e)  = [S,C3 

6([S],g)  = [S,D,E] 

$([S,D,E],f)  = [S,D,E,T] 

6([A3,c)  = [T] 

5 ( [S,D,E] ,g)  = [S,D,E,T3 

6([A],d)  = [T] 

5([S,A,T],c)  = [S,A,T] 

6([B],d)  = [T] 

«([S,A,T],d)  = [S,A,B] 

6([C],e)  = [I] 

6([S,A,T3,e)  = [S,C] 

«([D],f)  = [T] 

6([S,A,T],f)  « [S,D,E] 

6([D],g)  = [T] 

6([S,A,T}>g)  = [S,D,E] 

6([E],f)  = [T] 

6([S,A,B,T],c)  = [S,A,T] 

6([E],g)  = [T] 

6(tS,A,B,T],d)  = [S,A,B,T] 

5(fT],c)  = [<f>3 

6([S,A,B,T],e)  - [S,C] 

6([T],d)  = [<?>] 

= [S,D,E] 

6([T],e)  = [*] 

6([S,A,B,T],g)  = [S,D,E] 

6([T],f)  - I#] 

6(fS,C,T],c)  = [S,A] 

«([T],g)  = U3 

6([S,C,T],d)  = [S,A,Bl 

«([S,A],c)  = [S,A,T] 

5([5,C,T3fe)  = [S,C,T] 

«([S,A],d)  = [S,A,B,T] 

5([S,C,T3,f)  = [S,D,E] 

6([S,A],e)  » [S,C] 

S([S,C,T3,g)  = [S,D,E] 

6<IS,A],f)  = [S,D,E] 

6([S,D,E,T],c)  = [S,A] 

«([S,A],g)  » [S,D,E] 

6([S,D,E,T],d)  = [S,A,B] 

5([S,A,B],c)  = [S,A,T] 

6([S,D,E,T],e)  = [S,Cj 

fi([S,A,B],d)  = [S,A,B,T] 

6([S,D,E,T],f)  = [S,D,E,T] 

6([S,A,B],e)  = [S,C] 

S([S,D,E,T],g)  = [S,D,E,T3 

S([S,A,B3,f)  = [S,D,E] 

6({T],a)  =>  [$] 

5 { [S, A, B] ,g)  = [S,D,E3 

6(fT],b)  = [$] 

J 
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S([5,C],c)  = [S,A]  6([T],h)  = [$] 

6([S,C],d)  = [S,A,B]  fi([S,C],f)  = [S.D.E] 

6([S,C],e)  = [S,C,T] 

The  finite-state  automaton  for  highway  recognition  based  on  the  in- 
ferred grammar  (k=2)  is  as  follows:  M„=  (i,  (1,  6,  qQ>  F) . 

Z = {a,  b,  c,  d,  e,  f,  g,  h} 

Q = VH  U {T} 

q = S 

'o 

F = {T} 

6 = 6 (S  ,<;)  = A 
6 (S,d)  « B 
6 (S ,d)  = D 
S (S ,c)  = C 
6(S,e)  « E 
S(S,f)  = F 
6(S,g)  = G 
6(S,g)  = J 
6(S,f)  = H 
6(A,c)  = J 
6(A,d)  = J 
6 (B ,d)  = K 
fi(C,c)  = J 
fi(D,d)  = J 
5(D,c)  * J 
6(D,e)  = L 
6(E,e)  = L 


90 


6(F,f)  = M 
6(F,g)  = H 
6(F,e)  * L 
6(G,g)  « H 
6(H,f)  = N 
6(1,  g)  = N 
6(1,  f)  = H 
6 (J,c)  = T 
6 (J,d)  = T 
S(M)  = T 
6 (L,e)  = T 
6(M,f)  = T 
6(M,g)  - T 
6 (H,f)  » T 
6(H,g)  = T 
6 (T,a)  = <}. 

6 (T, b)  = $ 

6(T,c)  = 4 
6(T,d)  = 4 
6(T,e)  = $ 

6(T,f)  = $ 
o(T,g)  = <}> 

6(T,h)  = <j> 

The  corresponding  deterministic  finite-state  automaton,  of  the 
above  finite-state  automaton  from  the  inferred  granmar,  (k=2) , is  as 
follows:  M2'  = (r,  Q',  qQ',  F")  («2  = (E,  Q,  6,  qo,  F)). 
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Z'  - I = {a,  b,  ct  d,  e,  f,  g,  hJ 

Q,'  = Vtt  U {T,<j>}U  {{A,C]>[B,D],[F,f!]l[G>l3,[K,j}>[M,N]> 
N 


V = [S] 

F'  = [T] 

S'  : <S([S],c)  = [A,C] 
<5  ( [S] , d)  = [B,0] 
6([S],e)  = [E3 
6([S],f)  --  [F,H] 
'•  \ (S1  ,g)  = [G,l] 
'{{/  »c)  = [J] 
3UA],d)  = [J] 
6([B],d)  = [K] 
6(tc],c)  = [J] 
S([D],c)  = tJ] 
6([D],d)  = [J] 
<5([D],e)  = [L] 
S([E],e)  = [L] 
6([F],e)  = [L] 

s ( [f]  ,f)  = [M3 
6([F3,g)  - [M3 
«([Gj,g)  - [M3 
«([Hj,f)  = [M3 
«([!3,f)  » [M3 
<5([i3»g)  = [m3 
6([J],c)  =*  [T3 
6([J],d)  - [T3 


6([M3,g)  - [T3 
«(CM],f)  - [Tl 

fi([H],g)  = [T] 

5 ( [T 3 »a)  « [*3 

«5([T],b)  = [*] 
6([T],c)  = [$J 
5([T3,d)  - W 
«(lT],e)  - M 
6([T],f)  - W 
6([':],g)  - M 

5 ( [A,C  ] ,c)  « [J] 
6([A,C],d)  - [J] 
S([B»D],c)  « [J3 
5([B,D3,d)  = [K,J] 
«S([B,D],e)  = [L] 
S([F,H],e)  = [L] 
6([F,fi],f)  = [M,N] 
6([F,H],g)  = [M3 
6([G,f],f)  - [Ml 
fiUG,l],g)  « [M,M] 
«(tMl,c)  = [T] 
6(lM3,d)  » [T] 
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«([M,N],f)  = [T] 
6([M,N],g)  = ET] 
5([T],h)  = ($] 


6([K],d)  = [T] 

S([L],e)  = [T] 

6([H],f)  = [T] 

The  algorithm  of  highway  recognition  by  the  syntax-controlled  method  was 
implemented  and  the  experiments  were  performed  by  the  deterministic  finite- 
state  automata  Mj ' and  M2".  A - illustrative  example  is  given  here  to 
describe  the  operations  of  t:  is  algorithm. 

Example:  An  image  window  from  a LANDSAT  image  (visible  bands)  is  process- 
ed by  the  transformation  r*rocess  and  the  transformed  window  is  shewn  as: 


The  window  pattern  is  encoded  as  the  string  dddd.  The  finite-state 
automaton,  M^,  works  as  follows:  the  input  string  Is  analyzed  from  the 
starting  symbol  [S],  The  transition  rule  fi(fS],d)  =*  [B,D]  Is  chosen  by 
scanning  the  present  ctates  and  inputs  of  all  the  transition  rules  of 
M2".  Then  a searching  of  the  state  [B,D]  is  performed  on  the  present 
state  of  the  transition  rules  of  M^".  The  transition  rule  fi([B,D],d)  = 
[K,Jj  is  located  and  this  rule  leads  to  the  next  state  [K,J].  Thereafter, 
the  rule  6([K,J3-d)  = [T 3 is  applied  to  obtain  the  next  state  [T]  and  the 
partial  part  (ddd)  of  the  string  (dddd)  is  recognized.  The  rule  S([T],d) 

= [$]  is  successfully  found  in  the  set  of  transition  rules  of  Mj".  Thus, 
the  string  dddd  is  accepted  by  this  automaton.  The  window  pattern  is 
recognized.  The  pattern  is  a highway  which  lies  from  northwest  to  south- 
east. The  recognized  pattern  is: 
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The  experiments  of  highway  recognition  by  the  automaton  and 
were  conducted  on  the  same  set  of  images . The  experimental  results  from 
the  automaton,  are  more  accurate  than  those  of  Mj"  but  the  CPU  time 

is  slightly  longer  than  that  of  the  automaton,  Mj'*.  Since  the  grammar 
inferred  by  k=2  describes  patterns  more  precisely  than  the  grammar  infer- 
red by  k=l , the  slight  difference  in  CPU  time  between  these  two  automata, 
Mj”*  and  is  acceptable.  For  example,  the  CPU  time  for  automaton,  " 
on  highv/ay  recognition  of  a 96  x 96  LANDSAT  image  is  20  seconds  on  the 
IBM  360/67  computer.  The  automaton,  analyzing  the  same  image  takes 

21  seconds  on  the  same  computer.  Thus,  the.  grammar,  k=2,  is  suggested 
as  the  better  highway  grammar.  The  grammars  of  the  cases  k=3  and  k=4 
are  not  suggested,  because  of  the  increase  in  computer  processing  time 
to  infer  these  grammars  and  the  increase  in  CPU  time  for  the  image  recog- 
nition by  the  automata,  and 

3.2.3.  Experimental  Results  on  Highway  and  River  Recognition  from  LANDSAT 
Images 

The  computer  experiments  cn  highways  and  rivers  were  conducted  in 
different  areas.  The  highv/ay  recognition  by  the  syntax-controlled  method 
on  the  image  (Figure  3*3(a))  is  given  in  Figure  3*12.  The  area  of  Figure 
3.12  is  the  same  as  that  marked  on  Figure  3»3(d).  The  computer  result 
shows  that  highways  I-65  and  1-465  were  successfully  recognized.  The 


Figure  3.12.  Highway  recognition  result  by  the  syntax- 
controlled  method  on  Figure  3.3(a). 
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Figure  3.15(a). 


Satellite  Image  of  downstown 
Indianapolis,  Indiana. 
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Figure  3.15(b).  Commerciol/industrial  area  recognition  by 
syntax-controlled  method  on  Figure  3.15(a). 
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Figure  3.15(d).  Indianapolis  downtown  map  (marked  area 
corresponds  to  Figure  3.15(a)). 
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image  is  96  x 96  in  size.  The  CPU  time  for  highway  recognition  by  the 
syntax-directed  method  on  the  same  image  took  27  seconds  compared  with 
the  21  seconds  here.  The  river  recognition  was  performed  on  the  same 
image  shown  in  Figure  3.3(a).  The  results  of  river  recognition  by  this 
syntax-controlled  method  is  shown  in  Figure  3.13.  The  CPU  processing 
time  for  this  method  is  12  seconds  by  the  automaton  (k=2  case)  on  the 
96  x 96  image  compared  with  20  seconds  for  the  syntax-directed  method. 

3.2.4.  Experimental  Results  on  Bridge  and  Commerct =»!/!ndustrial  Area 
Recognition  from  LAI1DSAT  Images 

Figure  3-14  shows  the  bridge  recognition  result  by  the  syntax- 
controlled  method.  The  recognizer  is  automaton  M * (k=2  case).  The 
length  is  automatically  calculated  as  672  meters  which  Is  the  same  re- 
sult as  that  obtained  by  the  syntax-directed  method.  The  CPU  time  for 
bridge  recognition  by  the  syntax-controlled  method  is  39  seconds,  whereas 
the  syntax-directed  method  requires  a longer  CPU  time,  hZ  seconds.  The 
commercial/industrial  area  recognition  was  performed  by  the  syntax- 
controlled  method  on  an  image  of  the  downtown  area  of  Indianapolis, 

Indiana,  Figure  3.15(a).  The  result  is  shewn  in  Figure  3.15(b).  The 
size  and  center  of  the  downtown  area  v/ere  automatically  calculated  from 
the  96  x 96  image  frame.  The  computer  processing  time  of  the  syntax- 
controlled  method  for  this  experiment  was  43  seconds.  The  downtown  map 
of  Indianapol  is  is  provided  in  Figure  3.15(d).  The  urban  development 
information  extraction  is  shown  in  Figure  3.15(c). 

3.3.  COMPARISON  OF  THE  SYNTAX-DIRECTED  AND  SYNTAX-CONTROLLED  METHODS 

As  has  been  stated  earlier,  the  syntax-directed  and  syntax-controlled 


methods  we re  implemented  and  experiments  on  highway,  river,  bridge,  and 


* 


in?. 


commercial/industrial  area  recognl t Ion  were  conducted  on  the  IBM  360/67 
computer  In  the  Laboratory  of  Applications  for  Remote  Sensing  (LARS) . 

The  LOGICAL  programming  technique  v/as  used  in  both  methods.  Comparative 
studies  were  also  carried  out  without  LOGICAL  programming.  The  one  using 
logical  programming  saved  30%  of  the  CPU  time  used  by  the  other.  Con- 
cerning computer  memory  sapce,  there  Is  another  advantage  to  LOGICAL 
programming  in  that  every  pixel  of  the  transformed  image  takes  only  one 
byte  for  storage.  (Usually  each  pixel  takes  4 bytes  for  storage  of  in- 
teger). Therefore,  it  v/as  found  in  this  experiment  that  the  use  of  LOGICAL 
programming  saved  approximately  75%  of  the  memory  space  required  by  the 
program  requiring  four  bytes  per  pixel. 

As  stated  above  the  CPU  time  for  the  syntax-directed  method  for  high- 
way recognition  from  a 96  x 96  image  takes  approximately  27  seconds.  The 
CPU  tine  for  the  syntax-directed  method  for  river,  birdge,  and  commercial/ 
industrial  area  recognition  from  % x 96  images  takes  20,  42,  and  46 
seconds,  respectively.  The  syntax-controlled  method  was  implemented  by 
the  same  programming  techniques  as  those  used  for  the  syntax-directed 
method.  The  CPU  time  for  this  second  method  for  highway,  river,  bridge, 
and  commercial /industrial  area  recognition  from  the  96  x 96  image  were 
20,  12,  38,  and  4l  seconds,  respectively  (for  an  automaton  corresponding 
tc  a grammar  by  the  k=l  inference  procedure).  For  an  automaton  corres- 
ponding to  the  grammar  by  the  k=2  inference  procedure,  the  CPU  time  for 
highway,  river,  bridge,  and  commercial/industrial  area  recognition  of 
the  same  images  were  21,  12,  39  and  43  seconds,  respectively.  The  com- 
parative performances  of  the  syntax-directed  and  syntax-controlled 
methods  are  listed  in  Table  3.1.  It  can  be  seen  that  the  syntax-controlled 
method  processes  the  same  image  for  all  the  tasks  faster  than  the  syntax- 
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Table  3*1.  CPU  tine  performance  comparison  of  syntax- 
directed  and  syntax-controlled  methods. 


Highv/ay 

River 

Bridge 

Commercial/ 

Industrial 

Syntax-Directed 

27  sec. 

20  sec. 

42  sec. 

46  sec. 

Syntax- 
Control  led 

k»l 

20  sec. 

12  sec. 

38  sec. 

41  sec. 

Syntax- 
Control  led 
k=2 

21  sec. 

12  sec. 

39  sec. 

43  sec. 

} 


O 


directed  method.  As  for  the  accuracy,  the  syntax  directed  and  syntax- 
controlled  methods  have  the  same  level  of  high  accuracy  In  the  experi- 
ments . 

^ In  general,  the  syntax-directed  methods,  commonly  known  as  the 
template-matching  method,  has  the  advantage  of  faster  software  development 
time  because  the  implementation  is  less  complicated  than  for  the  syntax- 
controlled  method.  However,  the  disadvantage  of  the  syntax-directed 
method  is  that  the  recognition  depends  on  a set  of  limited  template 
patterns.  Once  the  pattern  does  not  natch  correctly  with  one  of  the 
templates,  then  the  pattern  is  rejected.  The  only  way  to  increase  the 
recognition  capability  is  by  adding  more  templates  in  the  set  of  tem- 
plate windows.  These  added  templates  increase  the  need  for  computer 
storage  and  the  CPU  time  for  each  matching  operation.  The  syntax- 
controlled  method  has  the  advantage  of  fast  computer  processing  time  for 
program  execution  once  the  more  complicated  programming  Is  completed 
which  means  a saving  of  CPU  time  in  processing  the  image  every  time 
compared  with  that  of  the  syntax-directed  method.  Another  advantage  of 
the  syntax-controlled  method  of  finite-state  string  grammar  is  that  the 
grammatical  inference  can  be  fully  computer  automated.  Thus,  the 
grammar  is  more  realistic  and  precise  in  describing  the  patterns  than 
is  the  gramnar  for  syntax-directed  method.  The  CPU  time  for  the  gram- 
matical inference  of  a highway  grammar  is  9 seconds  for  the  case  of 
k=l  and  15  seconds  for  the  case  of  k=2  in  the  k-tail  Inference  method 
of  the  syntax-controlled  method,  as  compared  with  the  grammatical  in- 
ference procedure  for  the  syntax-directed  method  in  which,  because  the 
procedure  is  an  interactive  process,  the  CPU  time  is  about  60  seconds. 
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In  conclusion,  the  syntax-controlled  method  for  highway,  river, 
bridge,  and  commercial /industrial  area  recognition  from  LANDSAT  images  is 
an  effective  technique  for  image  recognition.  The  results  from  such 
applications  can  contribute  to  urban  development  planning,  and  to  mili- 
tary reconnaissance. 
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CHAPTER  4 

IMAGE  SEGMENTATION  USING  A TREE  GRAMMAR  APPROACH 
The  syntactic  method  for  Inage  segmentation  was  developed  by  a tree 
grammar  approach.  The  reasons  that  a tree  grammar  approach  is  proposed 
for  image  segmentation  are;  first,  tree  languages  are  very  descriptive 
and  second,  the  tree  grammar  analysis  offers  a natural  hlgti-dimensional 

genera! tzat ten  of  strings.  Since  the  boundary  patterns  of  an  image  are 
usually  t wo  dimensional  and  tree  grammar  Is  more  convenient  for  describing 
high-dimensional  objects  than  a string  grammar,  the  tree  grammar  is  used 
to  describe  the  boundaries  of  the  homogeneous  segments  of  an  image,  in 
addition,  the  high  recogni tlon capabll 1 ty  of  tree  automaton,  corresponding 
to  the  tree  grammar,  and  the  hierarchical  nature  of  scenes  make  the  tree 
grammar  approach  very  attractive  for  image  segmentation. 

A syntactic  approach  to  Image  segmentation  was  investigated  which 
involves  two  levels  of  processing.  The  first  level,  referred  to  as  pre- 
processing and  primitive  extraction,  consists  of  two  steps  (l)  texture 
region  primitive  extraction,  and  (2)  boundary  primitive  extraction.  The 
second  level,  which  is  the  syntactic  analysis,  requires  tree  grammar 
inference  to  describe  the  boundaries  of  homogeneous  regions.  A block 
diagram  of  the  system  for  image  segmentation  is  shown  in  Figure  4.1. 

The  process  of  tree  grammar  analysis  utilizes  the  corresponding  tree 
automaton  from  the  Inferred  tree  grammar  to  process  the  primitive  extract- 
ed image.  Then  this  image  is  segmented. 
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Input  Image 


Sample  Patterns 
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Segmentation  Result 


Figure  >t.l.  Block  diagram  of  the  system  for  Image  segmentation. 
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4.1  SYNTACTIC  IMAGE  SEGMENTATION  ALGORITHM  (SISA) 

Several  approaches  to  the  use  of  texture  Information  in  Image 
analysis  have  recently  been  developed  [72,73].  However,  these  techniques 
have  r.  erally  been  applied  to  terrain  classification  following  segmen- 
tation and  not  to  the  segmentation  problem  itself.  In  our  approach  to 
image  segmentation,  the  texture  information  and  boundary  structures  are 
analyzed  to  obtain  the  image  segmentation  result.  The  input  to  the 
system  is  the  digitized  image.  The  texture  region  primitive  extraction 
performs  the  texture  analysis  and  extracts  the  texture  region  primitive. 
Then,  the  boundary  primitive  extraction  extracts  the  boundary  primitives 
from  the  result  of  the  texture  region  primitive  extraction.  The  result 
from  the  boundary  primitive  extraction  is  then  modeled  by  the  tree 
grammar  which  is  inferred  by  the  tree  grammar's  grammatical  inference 
procedure.  Each  part  of  the  image  segmentation  system  is  discussed  in 
detail  below. 

4.1.1  Inference  of  Tree  Grammars 

The  concept  of  tree  grammar  analysis  is  herein  described  by  defining 

and  studying  tree  grammars  and  tre^-  automata  [2], 

Definition  i: 

A pattern  grammar  G is  a four  tuple  G = (V^,  V^,  P,  S)  where 
Vfj  is  a set  of  non-terminals  or  subpatterns. 

VT  is  a set  of  terminals  or  pattern  primitives. 

Se  Vjj  is  the  start  symbol,  and 

Pisa  set  of  syntax  rules  or  productions  in  the  form  of  a -»•  fi. 
ac(V|(  U VT)*VH  (Vt|  U VT)\  MV(,  U 

V is  the  set  of  all  possible  sequences  of  symbols  In  V including 
the  empty  sequence  A. 
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By  applying  productions  in  P successively,  starting  from  S,  the 

strings  or  sentences  of  terminals  generated  by  the  production  In  P are 

•» 

the  language  g nerated  by  the  grammar  G,  L(G). 

Definition  2: 

Let  N+  be  the  set  of  strictly  positive  integers. 

Let  U be  the  universal  tree  domain  (the  free  semigroup  with  identity 
element  "0"  generated  by  N+  and  a binary  operation  [2,95]. 

Definition  3: 

A ranked  alphabet  is  a pair  <I,r>  where  Z is  a finite  set  of  symbols 
and  y:  E -*■  N — M U {0},  For  ae£»  r(a)  is  called  the  rank  of  a. 
Definition  l>: 

A tree  over  Z (i.e.,  over  <£,r>)  Is  a function  a : D ->  £ such  that 
D Is  a tree  domain  and  r[a(a)]  » max{i|a  * I e D}.  The  domain  of  a 
tree,  a,  is  denoted  by  D(ct).  Let  T^..  be  the  set  of  all  trees  over  £. 
Definition  5: 

Let  a be  a tree  and  a be  a member  of  D(a),  e[a,  a subtree  of  a at  a 
is  defined  as  a|a  ® { (b,x) | (a*b,x)  e o}. 

Definition  6: 

A regular  tree  grammar  Gt  over  <VT,r>  is  a grammar  Gt=(V,  r't  P,  S) 
satisfying  the  following  conditions: 

(i)  <V,r‘”>  Is  a finite  ranked  alphabet  such  that  C V and  r"[V^.  = 

r,  V.j.  and  V - are  the  same  as  in  Definition  1. 

(il)  P is  a finite  set  of  productions  of  the  form  $ ■*  $ where 
e Tv  (T^  is  a set  of  trees  over  <\l,r'>). 

(iil)  S Is  a finite  subset  of  T^. 

I 

f 

I 
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- Definition  7 

A tree  grammar  Gt  = (V,  r,  /,  S)  Is  expansive  If  and  only  if  each 
^ production  In  P is  of  the  form 


X -*■ 
o 


X 

x7  Xx 

X1  — r 


where  x e V^.  and  , X^,  ...»  e V^(-V-V^.)  are  non-terminal  symbols. 
Theorem  1 : For  each  regular  tree  grammar,  Gt,  one  can  effectively  con- 
struct an  equivalent  expansive  grammar  G^,  that  is,  L(Gt")  = 

L(Gt)  [2]. 


Definition  S 

A tree  automaton  over  Z is  a (k+2) -tuple 
Mt  " fl»  •••»  fk»  F) 

where  (i)  Q is  a finite  set  of  states;  (li)  for  each  i,  1 < ! < k, 

r(<0  r(ojJ", 

f.  is  a relation  on  Q x Q,  o}  e Z,  that  is  ff  : 0 ■*  Q; . _ 

and  (iii)  F = Q is  a set  of  final  states. 

Definition  9 

The  response  relation  P of  a tree  automaton  It  is  defined  as  (i)  if 
o e Zq,  p(o)  *=  X (X  belongs  to  final  state  set  F of  the  tree  auto- 
maton) if  and  only  if  f « X,  that  is,  p(o)  = f ; (H)  if  o e Z , 

o on 

n 0,  p(o,  X , ...,  X .)  — X (X  , ...,  X | are  non— terminals) . 
o n- I o n- i 

If  and  only  if  there  exists  X , ...,  X . such  that  f (X  , 

o'  ' n-l  . o o * 

Vi>  ■ x- 

Definition  10 

T(Ht)  **  (o  c T^  J there  exists  X e F such  that  p(a)  «=  X}  Is  called 
the  set  of  trees  accepted  by  It  . 


in 


( 
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Theorem  2:  For  every  regular  tree  grammar,  Gfc,  one  can  effectively  con- 
struct a tree  automaton  M such  that  T(Mt)  = L(Gt)  [2], 

The  construction  procedure  Is  summarized  as  follows: 

(!)  Obtain  an  expansive  tree  grarrmar  G * = (V't  r,  P",  S)  for 
the  given  regular  tree  grammar  G = (V,  r,  P,  S)  over 
alphabet  V^.. 

(II)  The  equivalent  (nondetemf nj stie)  tree  autonaton  Is 

Mt  *»  (Q,  fj,  ....  fk>  F)  where  Q.  - {Xq,  Xj,  ...,  Xn}=>  F 

fx<V  •••*  V = Xo  !f  Xo  x V •••»  Xn  's  'n  P'’  x e VT* 

X.,  X e Q,  and  f (X  ) - X ifX  x is  in  P".  x e V , 

« O X X o o I 

Xj , Xr  t Q,  F is  the  set  of  final  state.  F = £Xx [x.  e 

VT  and  fx(Xx)  = Xo). 

An  Illustrative  example  is  provided  here  to  show  the  procedure  of  con- 
structing the  tree  automaton.  The  tree  grammar  is  ■*  (V,  r,  P,  S) , 
where  V — {S,  a,  b.  A,  B},  Is  the  starting  node  or  root  of  the  tree, 
VT  - {a,  b,  $},  r(a)  * {2,0},  r (b)  = {0},  r($)  = {2},  and  P: 


/ \ 

A B 


a 

/ \ 
A B 


A -*■  a B -y  b 

The  procedure  of  construction  for  tree  automaton  Is  as  follows: 

(1)  Obtain  an  expansive  tree  grammar  G * =»  (V*,  r,  P%  S). 

Since  G^  Is  an  expansive  tree  grammar,  thus  G^  =»  « 

(V,  r,  P,  S)  for  this  example  grartnar. 

(U)  The  tree  autonaton  is  M *=  (Q,  fj,  ffl,  ffa,  F).  The  Q,  fj,  f#, 
f^,  and  F are  constructed  as  follows.  Since  the  granroar 
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rule  is  S ->  ^ * the  relation  is  obtained  as  f 

/\ 

A B 

The  relation  fg(qA,qB)  **  q^  Is  obtained  fron  grammar  rule 

A a , The  relation  f (q  ) = q Is  obtained  from  grammar 
/ \ a a A 

A B 

rule  A ->  a,  and  f^q^)  = qg  Is  obtained  from  grammar  rule 
B -»•  b.  The  tree  automaton  is  thus  constructed  as  M = 

{qs,  qA,  qg,  qa»  qb>»  the  relation  f*s  are  f^,  fg,  ffa,  and 
the  final  state  set  F - {q  ,q  }. 

3 b 

in  order  to  model  a language  more  realistically  or  to  describe  a 
class  of  patterns  more  precisely,  it  is  expected  that  the  grammar  used 
can  be  directly  inferred  from  a set  of  sample  sentences  or  a set  of 
sample  patterns.  This  subject  of  “learning”  a grammar  based  on  a set  of 
sample  sentences  is  called  grammatical  inference.  A tree  grammar  infer- 
ence procedure  is  briefly  reviewed  [201  bcl^w* 
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Represent  each  sample  tree  a.  as 


• 

where  any  of  the  subtrees  t,.t  t ; 

12’  n* 

consists  of  repetitive  substructures. 

(11)  Starting  from  the  root,  determine  subtrees  with  depth  one  for 

each  sample  tree  Uj,  excluding  the  subtrees  having  repetitive  sub- 
structures. 

(iii)  Attach  non-terminal  symbols  to  nodes  and  construct  an  expansive 
tree  grammar  G.  for  a.. 

(iv)  The  inferred  tree  grammar  for  the  complete  sample  set  will  then  be 


m 


G “ U G. 

* l ' 

The  purpose  of  the  inferred  tree  grammar  is  to  describe  the  boundaries 
of  the  image  segments.  The  primitives  for  those  patterns  are: 

8 I e .. 

b / f / 

c — - g 

d \ h \ 

and  the  window  size  is  chosen  as  an  0x8  array  of  pixels.  The  positive 
samples  are  those  patterns  starting  from  a primitive  fol lowed  by,  at  most, 
three  branches.  The  negative  samples  are  those  patterns  in  which  there 
( Is  no  boundary  line  or  those  having  singular  primitives  or  pixels.  The 

sample  patterns  are  listed  in  Appendix  A.  Applying  the  tree  grammatical 
inference  procedure  [20],  a set  of  tree  grammars  is  inferred  to  describe 
the  boundary  structures.  An  illustrative  example  is  given  here  to  show 
the  grammatical  inference  procedure  of  a tree  grammar.  For  example,  the 
sample  patterns  are  given  as  follows. 


E 

rm 

■J 

H 

S 


m 


The  grammatical  inference  procedure  works  as  follows:  (The  flow  chart  of 
the  tree  grammar  inference  procedure  Is  provided  In  Appendix  G). 


Step  1.  Represent  sample  trees  ^ and  ct„ 


c 

I 

c 

/ \ 
c e 

i 1 

c e 

l 
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a 


r 


as, 

d 

/ \ 
c e 
i 1 
c e 

i ' 

c e 


The  subtrees  of  and  a?  are  tj,  t2>  and  tg  for  ctj  and  tj', 


t't  and  t **  for  a„. 
2 s 2 


t 
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= d 
/ \ 
c e 
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Step  Determine  subtrees  with  depth  one. 
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si 


= d 
/ \ 
c e 
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Step  3* 


Attach  nonterminal  symbols  to  nodes  and  construct  an  ex- 
pansive tree  grammar  Gj  far  Oj  and  G^  for  a^.  (4  starting 

symbol) 


V 


s-M 


V1’ 


4 


A 


1 


A,  ■*  c 

1 I 

A 


1 


/\ 

A!  A2 


A,  + c 
/ \ 


'l 


A1  A2 


A_  -»■  e 
Z i 


A1  -*•  c 


A -*■  e 

A 


Step  b.  The  inferred  grammar  is  Gt  which  is  the  onion  set  of  the 
grammar  set  Gj  and  G2  of  Step  3. 

The  tree  grammar  is  obtained  from  the  grammatical  Inference  procedure.  The 


tree  grammar  Gf  Is  as  follows. 


nf> 


Gt  - (V,  r,  P,  S) 

where  V = {S,  4,  A,,  A.,,  Aj,  A^,  Afi,  A?,  a,b,c,d,e,f ,g,h} 

, r(a)  = {1,0}  r(b)  - {1,0}  r(c)  = {2,1,0}  r(d)  = {3, 2,1,0} 
r(e)  = 0,2, 1,0}  r(f)  = {2,1,0}  r(g)  = {2,1,0}  r(h)  = {1,0} 
r(4)  = {2,1} 

Vy  = {+  a,  ^ b,  ->  c,  \d,  I e,  ^f,  «-  g,  x h,  4} 
and  grar.nar  rules  P: 


S-*  4 S -) 


/ \ 
*2 


’ f ’•  t 


A,  e 

1 y x 

A,  A2 


2 / \ 
A,  A2 


'I 


A,  ->  c 
/ N 

A,  A2 


d 

/ \ 

A1  A2 


A,  ->  f 
1 / \ 

A1  A2 


A2  * 

A,  A2 


A.  **•  d 

/ I \ 

A3  A4  A5 


A -v  f 
2 / \ 

A,  A2 


A,  ->  e 

/IN 

A3  A*  A5 


A.,  -*■  b 

1 l 


A ->  d 
3 l 


A,  *»•  f 

->  I 


A ->  9 

3 I 

A. 


S -» 


I 

I 


I 
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A transformational  grammar  is  a set  of  grammatical  rules  for  trans- 
forming a pattern  from  one  form  to  another  [2,21],  A line  smoothing  tech- 
nique can  be  designed  by  a transformational  grammar.  Here  we  introduce 
the  tree  transformat iona . grammar  for  the  line  smoothing  technique.  The 
concept  of  the  syntactic  line  smoothing  technique  is  as  follows:  ir- 
regularities are  usually  caused  by  the  digitizer,  noisy  patterns,  and  so 
forth.  These  are  In  forms  such  as  the  zig-zagging  of  the  line  patterns. 
The  tree  transformational  grammar  evaluates  the  contextual  information  of 
the  patterns.  If  the  context  of  the  pattern  satisfies  the  transformational 
grammar,  that  pattern  is  transformed  into  a smoother  pattern.  By  this 


syntactic  line  smoothing  technique,  the  zig-zagging  of  lines  is  smoothed. 
Actually,  tree  transformational  grammar  Is  a universal  method  for  line 
smoothing  of  any  pictorial  data. 

The  process  grammatical  inference  scheme  for  tree  transformational 
grammar  is  as  follows: 

(1)  Represent  each  non-smooth  pattern  as 

! * 

*■ 

■ v 

where  tj  and  t2  are  the  predecessor  and  successor  subtrees  of 

the  t . subtree. 
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(2)  Based  on  the  relationship  of  tj,  and  t^,,  Interchange  t^. 

with  a smoother  pattern  t then  obtain  tree  transformational 

s i 

grammar  G.  for  t^j,  as: 


(3)  The  inferred  tree  transformational  grammar  for  the  complete 
sample  set  will  then  be  = U G^ 


For  example: 

non-smooth  pattern  (6x6)  $ 


smoothed  pattern  (6x6)  $ 


The  grammatical  Inference  scheme  for  tree  transformational  grammar  works 
as  follows. 

Step  1 . Represent  non-smooth  pattern  as 
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The  graphical  interpretation  of  the  transformational  grammar  Is  given  under 
each  rule  in  terms  of  non-smooth  and  smoothed  patterns. 
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<♦.1.2  Texture  Region  Primitive  Extraction 

The  texture  region  primitive  extraction  takes  the  digitized  image 
as  input  and  perform  texture  analysis  to  obtain  the  texture  region 
primitives.  This  process  consists  of  the  sub-processes  histogram 
equalization,  variability  texture  measurement,  ami  texture  region  primi- 
tive assignment. 

1)  Histogram  Equalization:  The  image  of  interest,  P(x,y),  is 

stored  as  an  H x M array  in  the  computer  memory.  Each  element  of  this 

set  has  a grey  level  value  given  by  the  intensity  function  z(x,y).  A 

histogram  of  an  image  is  defined  as  follows  [83):  Given  an  image  f,  let 

P^(z)  denote  the  relative  frequency  with  which  grey  level  z occurs  in  f, 

for  all  z in  the  g/ey  level  range  [z^ , z^]  of  f.  The  graph  of  P^(z)  as 

a function  of  z,  normalized  so  that  J^k  P^(z)dz  is  equal  to  the  area  of 

Z1 

f is  called  the  histogram  of  f.  The  histogram  equalization  technique 
[72]  requantizes  the  grey  levels  to  k**  levels  which  have  z^,  Z2,  ...»  z^' 
as  the  values  of  the  requantiz^cion  interval  points,  with  the  values  of 

I 
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z j and  Zj^  equal  to  the  minimum  and  maximum  values  of  Zj  respectively. 

The  histogram  equalization  technique  reassigns  values  of  grey  levels  to 
the  picture  pixels.  This  technique  assigns  an  equal  number  of  picture 
pixels  to  each  interval  of  the  grey  level  z^j...  Thus,  the  number  of 
grey  levels  of  the  original  digitized  Image  (usually  256  or  128  grey 
levels)  is  reduced  to  a reasonably  smaller  number  (e.g.,  16  or  8 grey 
levels).  Even  if,  for  example,  two  images  of  the  sane  scene  had  dif- 
ferent digitized  values  due  to  the  fact  that  they  v/ere  digitized  on 
different  machines  under  different  amounts  of  brightness,  the  histogram 
equalization  technique  would  assign  values  to  pixels  according  to  their 
relative  values  to  each  other,  and,  thus,  the  histogram  equalization 
results  of  these  tv/o  Images  of  the  same  scene  would  be  the  same. 

2)  Variability  Texture  Measurement.  Texture  information  is  ex- 
tracted from  the  spatial  relationship  of  the  grey  levels  of  pixels  in 
the  image.  The  joint  probability  density  of  the  pairs  of  grey  levels 
that  occur  at  pairs  of  points  with  distance  d is  calculated,  if  there 
are  k grey  levels,  the  array  is  a k x k matrix  P(i,j),  called  a co- 
occurrence matrix  [73].  Then  a variability  texture  feature  Is  calculated 
to  measure  the  spatial  relationship  of  the  grey  levels  of  an  image. 
Havalick  [72]  suggested  twelve  texture  feature  measurements  for  terrain 
classification.  Here,  v/e  use  a method  of  texture  analysis  for  image 
segmentation.  The  variability  texture  feature  is  the  modified  entropy 
texture  feature  [72].  (entropy  =*Z  Z log  (.'  • The 

« j 

logarithm  portion  (log  (HLJi), 

of  the  entropy  texture  feature  ?s 


R Is  the  normalization  constant  of  the 


matric  P(I,J).  K is  the  range  factor  to  expand  the  range  of  the  values 





of  the  texture  measurements.  The  variability  texture  Is  thus  defined 
as  [72]: 

VARIABILITY  « - E E (?JLl)-)K  log  , K > 1 


In  the  previous  research  work  in  texture  analysis  for  terrain  classifi- 
cation [72,733,  there  is  no  indication  which  texture  feature  is  the  best 
for  terrain  classification.  Therefore,  the  variability  texture  feature 
and  some  other  texture  features  such  as  second  order  moment,  contrast, 
and  correlation  are  measured  on  the  same  image  [723.  from  our  experi- 
ments on  images  of  several  areas  in  Indiana,  the  variability  texture 
measurement  Characterizes  the  major  land-use  classes  as  agricultural, 
wooded,  old  residential,  new  residential,  and  watery  areas.  Thus,  the 
variability  texture  measurement  is  used  for  image  segmentation.  The 
use  of  more  than  one  texture  feature  {In  addition  to  variability  texture) 
could  be  useful  to  image  segmentation.  The  reason  that  only  one  texture 
feature  is  used,  is  that  the  computer  processing  time  increases  as  more 
texture  features  are  calculated.  The  preprocessed  and  primitive  ex- 
traction result  is  to  be  processed  by  syntactic  analysis.  Hence,  one 
texture  feature,  variability,  is  used  in  the  preprocessing  and  primitive 
extraction.  Figure  4.2  is  an  illustration  of  the  variability  texture 
measurements  of  an  11  x 11  Image  window.  The  distance  is  one  for  the 
horizontal,  vertical,  left  diagonal,  and  right  diagonal  co-occurrence 
matrices.  The  variability  texture  Is  calculated  on  each  of  the  four 
matrices,  separately.  Th';  average  value  of  these  four  measurements  is 
taken  as  the  texture  measurement  of  the  center  (4x4  pixels)  of  that 
Image  window. 
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Figure  h. 
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. An  Illustrative  example  of  variability  texture  measurement 
on  an  11x11  image  window. 
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In  segmenting  a large  Image,  the  window  size  used  for  texture  measure- 
ment Is  11x11  pixels.  The  variability  texture  feature  Is  measured  from 
the  11x11  window  and  the  value  of  this  measurement  Is  the  texture  value 
of  the  center  4x4  cell.  This  technique  was  devised  because  it  Is  quite 
possible  for  a textural  area  to  be  smaller  than  the  window  size,  or  that 
the  boundary  between  different  textural  areas  lies  In  the  operation 
window.  In  this  proposed  technique,  such  a problen  Is  taken  care  of  as 
v/e  locate  the  boundaries  by  shifting  the  llxll  operation  window  4 pixels 
at  a time.  Thus,  the  potential  boundaries  could  be  preserved  and  the 
spatial  relationship  can  stiil  be  extracted  because  the  window  size 
(llxll)  has  not  been  reduced.  Several  other  sizes  of  operation  window 
such  as  9x3,  10x10,  12x12  and  13x13  v/e  re  also  tried.  The  texture 
measurements  from  the  smaller  windows  sometimes  failed  to  give  the  same 
good  results  as  did  those  from  the  llxll  window.  The  reason  for  this  is 
that  the  measurements  from  smaller  window  did  not  yield  enough  global 
information  of  the  image  data  to  allow  the  extraction  of  the  proper 
texture  measurements.  The  results  from  the  texture  measurements  of 
larger  windows  v/ere  similar  to  those  of  the  llxll  window.  The  compara- 
tive computational  time  for  these  measurements  by  different  windows  is 
that  the  time  for  a 9x9  window  is  about  5 % less  than  that  for  the  llxll, 
and  the  tine  for  the  13x13  is  about  10S  more.  From  this  study  of 
texture  measurements  with  respect  to  window  sizes,  the  llxll  window  was 
used  for  the  texture  measurements  in  our  experiments  of  the  LANDSAT 
images. 

3)  Texture  Region  Primitive  Assignment.  After  obtaining  the  texture 
values  for  4x4  unit  cells,  the  histogram  of  the  texture  values  In  the 
texture  domain  is  thresholded  and  then  assign  texture  codes  to  the 
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segments.  The  histogram  is  made  by  shifting  the  11x11  window  11  pixels 
at  a time,  because  only  the  global  distributions  of  the  texture  values 
are  of  interest  here.  The  feasibility  of  variability  texture  measure- 
ments and  this  technique  can  be  shown  as  follows:  Figure  4.3  is  the 
histogram  result  of  variability  texture  measurements  on  a 385x198  pixels 

LAMDSAT  image  of  an  Indiana  area.  Figure  4.4  is  the  histogram  result  of 

2 

angular  second  moment  (Angular  second  moment  = 2 Z (P(i,j)}  ).  Figure 

» J 

4.5  is  the  histogram  result  of  texture  neasu-ements  of  contrast  (Contrast  = 
Ng-1  ? Ng  Ng 

Z n { z £ P(i,j)}).  Figure  4.6  is  the  histogram  result  of  texture 

n=0  i=1  j=l 

i-j|-n 

measurements  of  correlation.  (Correlation  = (Z  E (i*j)P(i,j)  " yxjxyJ/ox 

» j . 

uy,  px,  py,  ax  and  ay  are  the  means  and  standard  deviations  of  Px  and  Py) . 
From  the  histogram  of  the  texture  measurements,  the  4x4  cells  are 
grouped  to  different  texture  regions. 

4.1.3  Boundary  Primitive  Extraction 

Following  the  texture  region  primitive  extraction  is  the  boundary 
primitive  extraction  consisting  of  horizontal  processing,  vertical  pro- 
cessing, logic  integration,  and  syntactic  line  smoothing. 

1)  Horizontal  Processing.  The  Horizontal  Processing  processes 
the  "texture  region  primitive  extracted  image"  row-wise  to  locate  the 
potential  horizontal  boundary  segments.  The  operation  procedure  Is  as 
follows:  let  Q.(  1 , J)  be  the  picture  function  at  location  (l,J). 

Step  1.  Start  with  Q(l,d)  as  reference. 

Step  2.  Compare  Q(I,J)  with  G(l,J+l).  if  the  distance  is  smaller 
than  a specified  value,  a "zero"  is  set  on  Q(t,J)  and 
Q(|,J+1).  Then  <l(l,J)  and  Q(l,J+2)  are  compared.  If  the 
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distance  is  greater  than  or  equal  to  the  specified  value. 

A "one"  is  set  for  Q(l,J+2)  as  a potential  boundary  primi- 
^ tive.  Then  the  same  process  is  applied  with  Q(l,J+3)  as 

the  reference. 

Step  3.  When  this  process  is  operated  to  the  rightmost  of  the  row, 
the  Q(!+l,  J)  is  the  reference  and  Step  2 Is  applied  until 
all  the  rows  are  processed. 

The  idea  of  this  process  is  to  treat  the  image  matrix  as  independent 
rows.  After  this  process  the  potential  vertical  boundaries  of  the  image 
are  detected.  The  reason  for  comparing  0(i,J)  and  0(l,J+2)  (when 
<l(l,J+l)=0)  in  Step  2,  is  because  the  reference  must  be  kept  In  the  sane 
operation.  If  instead  of  comparing  Q(I,J)  and  <i(l,J+2),  Q(I,J+1)  and 
Q(l,J+2)  are  compared.  The  reference  Is  shifted,  thus  none  of  the 
boundary  primitives  will  be  detected. 

2)  Vertical  Processing.  The  Vertical  Processing  is  similar  to  the 
Horizontal  once  except  that  it  processes  the  inage  column-wise  to  locate 
the  potential  horizontal  boundary  segments. 

3)  Logic  Integration.  The  result  of  horizontal  processing  Is  de- 
fined as  H and  the  result  of  vertical  processing  Is  defined  as  V.  The 
Logic  Integration  is  a boolean  "OR"  function  of  H and  V and  It  is  de- 
fined as  R(H,V). 

H V R(H,V) 

0 0 0 

0 1 1 

1 0 1 

1 1 1 
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The  software  program  Is  implemented  to  compare  H(|,J)  and  V(l,J)  to 
obtain  the  logic  integration  result. 

After  the  process  of  logic  integration,  the  potential  boundaries  of 
the  picture  are  detected.  The  deformation  of  the  boundaries  is  to  be 
removed  by  the  line  smoothing  technique  which  is  a syntactic  process  of 
transformational  grammar. 

4)  Syntactic  Line  Smoothing.  A tree  transformational  grammar  Is 
designed  to  reduce  irregularities  and  smooth  the  patterns.  The  set  of 
tree  transforms tiona!  grammar  for  syntactic  ’line  smoothing  is  inferred 
by  the  inference  scheme  in  section  4.1.1.  in  implementing  the  tree 
transformation,  first,  the  image  window  is  encoded  into  the  tree  language 
which  describe  the  boundaries  ?n  that  window.  The  output  of  the  logic 
integration  is  the  binary  Image  window.  The  encoding  procedure  checks 
the  window  row  by  row  to  find  the  first  pixel  with  non-zero  value  as  the 
starting  point  (^) . Then,  a search  for  surrounding  non-zero  points  is 
followed.  The  non-zero  points  are  encoded  as  the  primitives.  These 
primitives  are  the  2x2  blocks,  a t,  bs,  c ■*,  d\ , e +,  f g •<-,  and 
h f which  were  graphically  shown  in  section  4.1.1.  If  only  one  branch 
ts  found  from  the  starting  point,  then  a pointer  is  used  to  link  the 
connecting  primitive  of  this  branch  with  the  starting  primitive,  if 
two  branches  are  found,  then  two  pointers  are  used  to  link  the  encoded 
primitive  of  each  branch  respectively.  The  same  encoding  procedure  is 
applied  to  each  branch  to  obtain  the  tree  structure  of  the  primitives 
In  the  image  window.  This  tree  structure  is  thus  the  tree  language  for 
the  tree  grammar  analysis.  For  example,  for  an  input  widnow  from  logic 

i 

integration  process  as  !! 

if 


In  processing  the  tree  representation  of  the  Input  pattern  by  the  tree 
transformation  grammar,  the  rule  (1) 

* * 

i l 

I I 

e ■+  d 

I /\ 

c c e 

I 

e 

/ \ 
c e 

will  produce  the  transformed  pattern  and  its  data  representation  is  as 


follows. 
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In  order  to  apply  the  tree  transformation  grammar  to  smooth  the  pattern, 
the  pattern  primitives  around  the  pattern  primitive  of  interest  have  to 
be, checked.  In  a 6x6  window,  there  are  eight  primitive  windows  (2x2) 
around  the  center  primitive  window  of  interest.  Thus,  there  is  enough 
information  for  deciding  whether  the  proper  smoothing  rule  will  be  applied 
to  smooth  the  pattern  or  not.  Therefore,  in  processing  an  image,  the 
syntactic  line  smoothing  process  is  performed  on  a 6x6  window.  The 
window  is  shifted  4 pixels  at  a time  to  repeat  the  process.  There  are 
always  2 pixels  overlapping  on  each  process,  thus  the  irregularities 
between  two  neighboring  windows  will  also  be  processed  by  this  line 
smoothing  technique.  When  the  window  operation  reaches  the  edge  of  the 
image,  it  moves  downward  four  pixels  and  starts  from  the  leftmost  point 
of  the  image  to  repeat  the  same  process,  until  the  whole  image  Is  pro- 
cessed by  this  syntactic  line  smoothing  technique. 

4.1.4  Tree  Grammar  Analysis 

The  recognition  by  tree  grammar  analysis  is  performed  by  the  tree 
automaton  corresponding  to  the  tree  grammar.  The  result  of  boundary 
primitive  extraction  Is  encoded  window  Ly  window  as  tree  structure  by 
the  encoding  procedure  described  in  section  4.1.3.  if  the  input  tree 
structure  can  be  derived  by  the  transition  rules  of  tree  automaton  to  the 
final  states,  then  this  tree  is  accepted  by  the  tree  automaton.  If  the 
input  tree  structure  cannot  be  derived  to  the  final  states  by  the  tree 
automaton,  then  the  input  structure  is  rejected. 

By  continually  performing  this  procedure  on  each  Image  window,  the 
boundary  structures  of  the  image  are  analysed  ana  the  syntactically 
correct  boundaries  of  image  segments  are  obtained,,  Hereby,  the  image 
segmentation  result  is  achieved. 
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Tree  Gramnar  Analysis  Algorithm 

input:  Boundary  Primitive  Extraction  Result,  Binary  image,-  p(l,j). 
Output:  Image  Segmentation  Result. 

Algor i thm: 

1.  Construct  the  tree  automaton  from  the  Inferred  tree  grammar 
of  the  boundary  structures!  The  procedure  of  tree  automaton 
construction  has  been  described  in  section  4.1.1. 

2.  initialize  the  array  P ( I , J)  as  the  operation  window  (8x8). 

Load  the  data  of  image  segment  H(f,J)  where  J =*  1,8  and 

I = 1,8  onto  the  operation  window  P(|,J). 

3.  Encode  the  array  P ( I , J)  into  a tree  structure  as  described 
in  the  section  on  syntactic  line  smoothing. 

4.  Each  transition  rule  of  tree  automaton  is  stored  in  computer 

as  a linear  array.  For  example,  ~ is  stored  as  a 

1x3  array  called  TA(i,J),  then  TA(l,l)  = qr  (present  state), 
TA(l,2)  = $ (input  state),  and  TA(l,3)  = q_j  (next  state). 

The  input  is  stored  as  a linked  tree  data  structure.  The 
root  ($)  of  the  tree  is  the  input  symbol  and  the  initial 
state  (present  state)  is  qg,  thus,  a next  state  Is  obtained 
by  searching  for  the  transition  rule,  from  the  array  TA(l,J), 
with  the  present  state  and  input  symbol  The  recognition 

of  each  branch  of  the  input  tree  is  similar  to  that  of  a 
finite  state  automaton.  If  all  the  branches  of  the  Input 
tree  can  achieve  the  final  states  by  applying  the  transition 
rules  of  the  tree  automaton,  then  the  input  tree  is  accepted. 
If  there  is  one  or  more  branches  which  cannot  achieve  the 
final  branches  which  cannot  achieve  the  final  states  bv 


mo 

checking  all  the  transition  rules;  of  the  tree  automaton, 
then  this  input  tree  Is  rejected.  The  syntactically  correct 
/"  boundary  patterns  are  thus  accepted  as  the  image  segmentation 

result. 

5.  Shift  four  columns  to  the  right  of  M(l,J).  Then  go  to  Step 
3 until  the  shifting  operation  reaches  the  rightmost  column. 
Go  to  Step  6. 

6.  Shift  four  rows  down,  go  to  Step  3 until  reaching  the  last 

row  of  the  image  M(l,J).  Then  the  image  is  analyzed 

by  the  tree  automaton  and  the  syntactically  correct  boun- 
daries are  accepted  as  the  image  segmentation  result.  The 
flow  chart  of  the  algorithm  is  given  ?n  Appendix  K. 

For  example,  using  the  grammar  inferred  in  the  example  of  grammati- 
cal Inference  in  section  4.1. I,  the  tree  grammar  is  as  follows: 

Gt  » (V,  r,  P,  S) 

V 53  C A | , ^2»  S,  ![,  c,  d,  e,  f} 

V.j.  = {|,  ->c,  d,  +e, 

r(s)  - {0,  r(c)  « (0,1,2},  r(d)  = {2},  r(e)  = {0,1}, 
r(f)  » {0,1} 


P:  s -*■  $ s -v  ^ 


The  Input  pattern  is  encoded  as 


\h\ 


The  input  tree  structure  is 


— ■ c 


T“"  * 
\ ' , 


As  described  In  section  ^.1.1,  a tree  automaton  can  be  constructed 

corresponding  to  the  tree  grammar  of  this  example.  The  tree  automaton 

is  thus  constructed  as  M.,  M.  = (Q,  f , f , f ,,  f»>  F) , where  Q = 

t t c e a $ 

{q  . ne»  <i,»  q?»  qs>.  f = {qc,  qe>,  and  f:  (The  detailed  steps  of  tree 
automaton  construction  are  given  in  Appendix  I)* 


(1) 

yv 

= qs 

(2) 

y^ 

= qs 

(3) 

yv 

12) 

Vi) 

fd(Vq2}  = 

(5) 

fc(V 

= q! 

(6) 

fe(<l2> 

- q2 

(7) 

yy 

" q1 

(8) 

feV 

= q2 

1 hz 

After  constructing  the  tree  automaton,  the  recognition  of  an  input  tree 
can  be  performed  by  the  tree  automaton.  The  recognition  of  the  input 
tree  of  this  example  Is  as  follows:  the  root  of  the  input  tree  is 

The  first  transition  rule  of  the  automaton  is  applied  to  obtain  the  next 

state  q^.  The  second  level  of  the  Input  tree  is  d.  The  transition  rule, 

which  has  d as  input  and  as  present  state,  cannot  be  found.  Thus, 

the  second  transition  rule  should  be  used  to  obtain  the  next  state  q2 
in  the  recognition  of  the  first  level  of  the  input  tree.  The  second 
level  of  the  input  tree  is  d.  The  transition  rule  with  present  state 
q2  and  input  d,  which  is  transition  rule  (*s) , produces  the  next  state 
pair  (qj,q2).  The  third  level  of  the  input  tree  is  branches  c and  e. 

With  present  state  q^  and  input  symbol  c,  the  transition  rule  (5)  pro- 
duces the  next  state  qp  The  fourth  level  of  the  input  tree  is  c.  The 
transition  rule  (5)  Is  applied  again  to  produce  the  next  state  The 

final  level  of  the  Input  tree  is  c.  The  transition  rule  (7)  Is  applied 

to  achieve  the  final  state  q . The  branch,  d,  of  the  third  level  of 

c 

the  input  tree  Is  recognized  by  transition  rules  (6),  (6),  and  (8)  for 
the  fourth,  fifth  and  final  level  of  the  input  tree  to  achieve  the 
final  state.  Thus,  all  the  branches  of  the  input  tree  achieve  final 
states,  then  the  tree  is  accepted  by  the  tree  automaton. 

ij.2  COMPUTER  EXPERIMENTAL  RESULTS  OF  IMAGE 
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Applying  the  grammatical  inference  procedure  of  tree  grammar  in 
section  4. 1.1,  a tree  grammar  was  inferred  to  describe  the  boundaries  of 
image  segments.  This  grammar,  as  stated  in  section  4.1.1,  was  applied 
to  the  image  segmentation  of  LAUDSAT  images.  The  tree  automaton  cor- 
responding to  the  tree  grammar  is  constructed  as  H^.  = ((),  fg,  f^. 


V fd*  fe’  ff’  fg’  fh’ 

f^,  F) . Where 

Q = {qj»q2»  q3>q2t»q5»q6 

,q7,qa’WVqe 

,qf’qg’VqS} 

F = {qa,  qb,  qc,  qd,  qfi 

• V qg*  qh5 

Vqi,q2)  “ qs 

f4(V  ’ qs 

f|(q6)  - q5 

f4  (t*z)  " qs 

VV  ’ qs 

fc(q,.q2)  - q 

fc(Vq2)  “q2 

fc(q1>q2)  ‘ql 

fd(Vq2!  “ q 

ff(q,,q2)  = q, 

fgtql’q2)  *q2 

VWV  ' 

f.*q3’Vq5*  = qi 

ff(q1*q2>  q2 

fd(q3)  = q3 

fe(q3>  - q3 

f,(q3)  - q3 

fg(q3>  “ q3 

W “ q4 

W ” q4 

fa(qi i>  = q4 

fciq3)  ” q3 

fh(q3)  - q3 

fb  V “ q4 

ff  ('ii,)  * qi. 

f g * 'll, 

fcV  ” % 

W " q4 

W = q5 

W " q5 

fe(q5>  " qS 

fe(qi,)  - q, 

VV  - q5 

Vq5>  ’ % 

W " q5 

fh(q5)  _ qs 

fe(q5>  = q5 

f.(q|>  ■ % 

fc<q7)  - q7 

fc(q,)  - q7 

W “ ql 

fc(ql>  = ql 

ff(q,)  » q, 

144 

fb(q2>  - q2 

f-A>  - q2 

W " q2 

fg(q2)'“  q2 

- q2 

/ 

W ■ q3 

,c(qc)  " ql 

ff{qf)  - qi 

W ■ q2 

fc(qc)  - q2 

fd<qd>  “ q2 

fg(qg>  “ q2 

W = q2 

W “ q3 

W = q3 

W - q3 

ffV  ” q3 

W - q* 

W " q« 

W = % 

ff(qf>  - % 

w = q* 

W “ q4 

W = q4 

fbCqb>  " q5 

fdtqd)  ' q5 

fe(qe>  " q5 

W - q5 

W " q5 

fc(qc>  = q5 

fh(qb)  ’ q5 

fetqe)  ” q6 

fc(qc>  “ q7 

W ” % 

fe(qe)  - q, 

j 

fb(qb>  ’ ql. 

fc(qcJ  ” q3 

fh(qh)  “ q3 

Figure  4. 7 Is  a LAMDSAT  image  of  Bloomington,  indtana  (88x08  In 
size).  Figure  4. 0(a)  Is  the  Intermediate  result  after  the  texture 
region  primitive  extraction  of  the  proposed  image  segmentation  method. 
The  syntactic  image  segmentation  algorithm  achieved  the  segmentation  of 
the  picture  in  Figure  4.7.  The  r^ul  t is  shown  In  Figure  4.8(b).  For 
the  purpose  of  knowing  the  ground  truth  of  this  satellite  image,  this 
area  has  been  classified  by  a maximum-likelihood  point-by-point  classi- 
fier. This  classification  result  is  shown  in  Figure  4.9  in  which  A 
stands  for  agricultural,  T for  forest,  X for  old  residential,  Y for 
new  residential,  and  V for  watery  areas.  The  computer  processing  time 
of  the  syntactic  method  is  only  55  seconds.  But  the  classification 
technique  takes  240  seconds  of  CPU  time. 
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Consider  now  the  performance  of  the  syntactic  Image  segmentation 
algorithm.  For  example,  the  left  upper  corner  (line  1-4,  columns  12-24) 
of' Figure  4.9,  the  ground  truth  of  the  image  Figure  4.7,  Is  a small  area 
of  foiest  area.  This  area  is  well  segmented  by  the  proposed  method  as 
a homogeneous  region  shown  in  the  corresponding  coordinates  in  Figure 
4.8(b).  One  good  thing  about  this  algorithm  is  that  in  the  segmentation 
result,  the  boundaries  of  the  inage  are  always  closed.  A closed  boun- 
dary of  image  segments  contributes  to  the  future  recognition  of  either 
the  shape  or  the  characteristic  pattern  of  that  segment. 

4.3  COMPUTER  EXPERIMENTAL  RESULTS  OF  IMAGE  SEGMENTATION 
FROM  FUR  ( FORWARD  LOOK  I HG  I NFRAREP)  TmAGET 

The  first  step  of  an  object  detection  problem  ts  actually  an  image 
segmentation  problem.  The  object  of  Interest  must  be  first  extracted 
from  the  scene.  }n  the  human  visual  system,  the  object  is  detected  by 
the  human  eye  through  the  characteristic  contents  and/or  the  shape 
of  the  object.  The  syntactic  Image  segmentation  algorithm  described  in 
section  4.1  tends  to  simulate  the  human  visual  system  for  object  detec- 
tion. The  texture  analysis  of  the  preprocessing  extracts  the  characters 
tic  contents  of  the  image.  The  syntactic  analysis  examines  the  boundary 
of  the  object.  The  syntactic  image  segmentation  algorithm  is  also 
adaptable  to  object  detection  from  FUR  Images.  When  the  image  Is  rich 
In  texture,  the  syntactic  method  for  image  segmentation  described  in 
section  4.1  can  be  applied  to  object  detection.  When  the  image  is  not 
rich  in  texture,  the  texture  feature  measurement  of  the  preprocessing 
part  of  the  syntactic  method  Is  changed  to  a mean  vector  measurement. 

The  syntactic  analysis  Is  unchanged. 
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Figure  4.8(d)  Result  of  syntactic  image  segmentation  of  Figure  4.7. 
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Figure  ^,3. 


Point-by-point  classification 


result  of  Figure  4.7. 
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If  several  objects  have  the  same  texture  contents,  but  different 
shapes,  then  the  object  of  Interest  Is  primarily  distinguished  from  the 
other  objects  by  the  second  process.  In  this  case  the  first  process 
contributes  to  the  extraction  of  those  objects  from  their  background. 

The  syntactic  analysis  process  examines  the  boundaries  of  these  objects 
and  accepts  those  objects  with  the  boundaries  which  can  be  generated  by 
the  grammar. 

If  several  objects  have  the  same  shapes,  but  different  texture  con- 
tents, the  first  process  In  the  syntactic  method  extracts  the  texture 
contents  of  the  objects.  If  the  contents  are  the  sane  as  those  of  the 
object  of  interest,  then  the  syntactic  analysis  examines  the  boundary  of 
the  object.  If  the  syntactic  analysis  has  a successful  parsing,  then  the 
object  of  interest  is  still  distinguished  from  other  objects  in  the 
image  scene. 

This  method  can  also  be  applied  to  tactical  target  detection  from 
infrared  images  of  battleground  scenes.  The  technique  used  here  is  the 
same  as  that  which  has  been  described  In  section  4.1  basically. 

4.3.1  Data  Acquisition  System  of  Infrared  Images 

The  infrared  images  were  obtained  from  Honeywell  Systems  and  Re- 
search Center  [ll6]  under  a consulting  contract.  The  Honeywell  Corpora- 
tion obtained  the  FUR  Imagery  using  Navy  Pi-V  aircraft  at  the  Naval  Air 
Test  Center  in  Patuxent  'Uvor,  Maryland,  and  a Honeywell  18  detector 
serial  scan  FUR  sensor.  The  P2-V  is  outfitted  as  an  electro-optical 
test  bed  allowing  the  collection  and  recording  of  taped  imagery  of  various 
kinds.  The  infrared  images  were  obtained  in  June  of  1974  and  were  of 
military  vehicles  at  Camp  A P Hill  In  southern  Maryland.  Altitudes  during 
the  flight  were  about  3,000  feet. 
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The  proposed  technique  of  image  segmentation  has  also  been  tested 
on  a different  data  set  of  infrared  images  which  were  collected  by  the 
flight  Vision  Laboratory.  Figure  4. 10(a),  4.13(a),  4.14(a)  and  4.15(a) 
are  the  test  images  from  Honeywell  and  Figures  4.11(a),  4.12(a),  4.16(a), 

4.17(a),  and  4.18(a)  are  the  test  images  from  the  Night  Vision  Laboratory. 

4.3.2  Experimental  Results 

Since  infrared  images  are  thermal  images,  the  characteristics  of 
these  images  are  different  than  those  of  the  LANDSAT  images.  Also,  the 
objects  of  interest  in  the  object  detection  from  infrared  images  are 
tactical  targets,  larger  one?  and  smell  ones,  such  as  military  vehicles, 
basically,  the  algorithm  for  the  experiments  on  infrared  images  is  the 
same  as  the  a 1 gor i then  described  in  section  4.1.  The  differences  are 
the  v/indow  size  for  the  texture  measurements  and  a different  tree  grammar 
1 to  describe  the  boundaries  of  military  vehicles.  Experiments  on  different 

window  sizes  for  texture  measurements  were  carried  out.  A window  size  of 
8x8  provided  the  best  segmentation  result  In  the  data  set  used.  Smaller 
v/indow  sizes  tended  to  give  segmentation  results  of  the  local  property, 
and  larger  windows  took  more  CPU  time  while  providing  no  better  results. 

Therefore,  the  window  size  used  for  texture  measurements  in  our  experi- 
ments on  infrared  images  was  8x8. 

The  texture  region  primitive  extraction  and  boundary  primitive  ex- 
raction  we re  the  same  here  as  those  in  section  4.1.  The  tree  grammar 
for  describing  the  boundary  structures  of  military  vehicles  is  inferred 
by  applying  the  tree  grammar  Inference  procedure  in  section  4.1.1  to  the 
sample  patterns  of  the  boundary  structures  of  military  vehicles  (Appendix 
F) . The  resultant  tree  grammar  Is  G^: 
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Since  the  shapes  of  the  objects  of  Interest,  military  vehicles.  In 
the  Image  segmentation  of  FUR  images  are  simple,  most  of  the  grammar 
rules  happen  to  be  string  (regular)  grammar  rules. 

The  tree  automaton  corresponding  to  the  tree  grammar  Is 

Mt  = (°-»  fa»  fb’  fc’  fd»  fe»  ff>  V V F* 

where 

Q s ^»Vq3’VVVVVVVVVVqs} 

F - (qa,  <lb,  qc,  qd>  qe>  V V V 

f^q,)  = q5  ff(q5)  " q5 
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VvV  ” q5 
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Figure  4.10(a)  is  an  Infrared  image  of  a tactical  target  scene. 

The  size  of  the  image  is  88x88.  The  altitude  of  the  infrared  sensor 
from  the  ground  was  about  3,000  feet.  The  window  size  used  for  texture 
measurements  was  8x8.  The  syntactic  analysis  was  performed  by  the  tree 
automaton,  corresponding  the  tree  grammar  for  describing  the  boundaries 
of  military  vehicles  including  tanks,  trucks,  and  armed  personnel 
carriers.  The  proposed  syntactic  method  successfully  segments  the  In- 
frared Image  Into  target  and  background.  The  Image  segmentation  result 
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Is  stiown  In  Figure  4.10(b).  The  symbol  T represents  the  boundary  of 
the  tactical  target.  (This  image  scene  is  a side  view  of  a track  heading 
east.)  The  confutation  time  for  Imaae  segmentation  of  Figure  4.10(a)  - 
Figure  4.18(a)  is  about  50  seconds  on  the  IBM  360167  computer.  Several 
experiments  on  texture  analysis  of  the  distinction  between  background 
and  targets  are  conducted.  Figure  10(c)  is  a segment  of  Figure  4.10(a). 

(Column  21-28,  row  61-68  corresponding  to  the  frame  88x88  of  Figure  4.10 
(a)).  The  texture  region  primitive  measurement  result  Is  in  Figure  4.10 
(d)  which  is  a segment  of  background  scene.  Figure  4.10(e)  is  a segment 
(Column  53-60,  row  61-68)  of  Figure  4.10(a).  The  texture  region  primi- 
tive measurement  result  is  in  Figure  4.10(f).  The  results  of  the 
similar  experiments  on  Figure  4.11(a)  are  provided  from  Figure  4.11(c) 
to  Figure  4.1 1 (f) . 

Figure  4.11(a)  is  an  .infrared  image  which  Is  a side  view  of  a tank. 

The  proposed  syntactic  method  was  applied  to  this  image.  The  tactical 
target  was  successfully  segmented.  The  segmentation  result  of  Figure 
4.11(a)  is  given  in  Figure  4.11(b).  The  sizes  of  the  images  from  4.11(a) 
to  4.18(a)  are  all  88x88.  Figure  4.12(a)  is  an  infrared  image  of  a 
vehicle,  and  the  result  of  Its  syntactic  image  segr.entation  is  shown  in 
Figure  4.12(b).  Figure  4.13(a)  is  a top  view  of  a vehicle.  The  target 
was  successfully  segmented  even  though  it  is  a noisy  image  of  low  reso- 
lution. The  result  Is  In  Figure  4.13(b).  The  image  segmentation  results 
from  Figures  4.14(a),  4.15(a),  and  4.16(a)  by  the  proposed  syntactic 
algorithm  are  given  in  Figures  4.14(b),  4.15(b),  and  4.16(b)  respectively. 
Experiments  involving  segmenting  snail  targets  were  also  conducted  by 
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Figure  A. 10(a).  Infrared  Image  of  a tactical  target  scene. 


Figure  4.10(b).  Image  segmentation  result  by  the  syntactic  method 
on  Figure  4.10(a). 
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Figure  4.10(c).  Image  segment 
8/8  of  the  background  of  Figure 
^ - 1 0 (a) . 
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Figure  4.10(d).  Texture  region 
primitive  measurement  result  of 
Figure  4. 10(c). 


Figure  4.  10(e).  Image  segment 
8x8  of  the  target  area  of  Figuj 
4.10(a) . 
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Figure  4.10(f).  Texture  region 
primitive  measurement  result  of 
Figure  4.10(e). 


Figure  4.11 (a) . Infrared  Image  of  a tactical  target  scene. 


Figure  4.1 l 


Image  segmentation  result  by  the  syntactic  method 
on  Figure  4. I l (a) . 
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Figure  A,  11(c).  Image  segment 
8x8  of  the  background  of  Figure 
A.  11(a). 
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Figure  A. 11 (d).  Texture  region 
primitive  measurement  result  of 
Figure  A. II (e) . 


Figure  A. 11(f).  Texture  region 
primitive  measurement  result  of 
Figure  A, 11 (e). 


Figure  4. 12(a).  Infrared  Image  of  a tactical  target  scene. 


Figure  4.12(b).  Image  segmentation  result  by  the  syntactic  method 
on  Figure  4.12(a) . 
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Figure  4.  13(b).  Image  segmentation  result  by  the  syntactic  method 
on  Figure  4.13(a). 


i 

I 


Figure  4.15(a).  Infrared  Image  of  a tactical  target  scene. 


Figure 


.15(b).  Image  segmentation  result  by  the  syntactic  method 
on  Figure  4. 15(a) . 


Figure  A. 16(a).  Infrared  Image  of  a tactical  target  scene. 


Figure  h. 16(b),  Image  segmentation  result  by  the  syntactic  method 
on  Figure  h. 16(a) . 


Figure  4..18(a).  Infrared  Image  of  a tactical  target  scene. 


Figure  4.18(b).  Image  segmentation  result  by  the  syntactic  method 
on  Figure  4.l8^b). 
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the  proposed  syntactic  algorithm  by  processing  the  infrared  Images  of 
Figures  4. 17(a)  and  4. 18(a).  Tfee  image  segmentation  results  achieved 
are  shown  in  Figures  4.17(b)  and  4.10(b). 

Comparing  the  segmentation  result  in  Figure  4.10(b)  with  the  test 
image  Figure  4.10(a),  the  object  of  interest,  a truck,  is  well  segmented. 
This  experiment  examp] if jes  the  performance  of  the  proposed  method.  The 
boundary  contour  of  the  object  of  interest  in  Figure  4 M(b)  can  be  used 
for  shape  recognition.  In  the  experiment  of  image  segmentation  on  Figure 
4.11(a),  a good  segmentation  result  of  the  te:  t image  fs  also  achieved. 
The  objects  of  interest  in  Figure  4.10(a)  and  4.11(a)  are  the  targets 
which  are  larger  than  most  of  the  targets  in  Figure  4.12-Figure  4.19. 
Thus,  the  image  segmentation  results  have  enough  boundary  information 
of  the  targets  for  the  future  recognition  purpose,  because  of  the 
low  resolution  of  the  images  in  Figures  4.12(a),  4.13(a),  4.14(a) 

4.15(a),  4.17(a),  and  4.18(a),  the  types  of  the  targets  carnot  be  told 
even  by  human.  It  is  hard  to  distinguish  them  by  the  segmentation 
result  either  * but  those  targets  are  still  segmented.  The  noises 
usually  are  caused  by  heat  diffusion  of  the  object  and  the  sensitivity 
of  the  infrared  sensor  and  these  noises  sometime  effect  the  segmentation 
result.  In  general,  the  syntax  ric  Image  segmentation  algorithm  still 
well  segments  the  image  into  target  and  background.  This  contributes  to 
the  information  extraction  from  Infrared  images. 

4.4  SUMMARY 

This  chapter  presents  a syntactic  method  for  image  segmentation.  It 
Is  a syntax-controlled  method  which  utilizes  a tree  automaton  to  extract 
the  boundaries  of  the  homogeneous  region  segments  of  an  image.  The 
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homogeneity  of  the.  segments  is  measured  by  texture  measurements  described 
in  section  4.1  .2.  r 

' It  has  been  shown  from  both  simulation  and  experimentation  that  the 
proposed  syntactic  method  segments  small  tefctural  areas  as  well  as 
larger  ones.  Thus,  this  method  contributes  to  the  automation  of  image 
understanding.  As  described  In  section  4.1,  the  best  v/indow  size  of  the 
texture  feature  measurement  for  the  proposed  method  needs  to  be  determined 
through  experiments  using  different  windows  on  a test  data  set.  Some 

texture  analysis  technique,  such  as  syntactic  texture  modeling  and  dis- 
crimination [117],  could  be  exploited  to  further  refine  the  texture 
region  primitive  extraction  of  the  proposed  algorithm. 
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✓ 

CHAPTER  5 

DESIGN  FOR  A SPECIAL  COMPUTER  ARCHITECTURE  FOR  IMAGE  PROCESSING 

Image  processing  by  computer  encompasses  a wide  variety  of  tech- 
niques and  mathematical  tools,  in  most  image  processing,  large  computers 
have  been  employed.  Unfortunately,  the  cost  ts  high.  As  pointed  out  in 
the  introduction,  image  processing  tasks  usually  involve  an  extremely 
large  volume  of  data,  much  of  which  can  be  operated  on  in  parallel. 
Therefore,  it  is  important  to  study  special  computer  archi tectures  for 
image  processing.  During  the  last  two  decades  the  field  of  image  pro- 
cessing has  grown  up  rapidly.  New  techniques,  algorithms,  and  applica- 
tions have  been  developed,  but  there  is  still  a need  for  improved  hard- 
ware. A special  computer  architecture  is  presented  here  as  a 
proposal  for  improving  the  state-of-the-art  in  image  processing  and  also 
to  cut  costs.  Designing  this  computer  architecture  v/as  a challenging 
problem,  as  the  desire  was  to  build  a computer  that  v/ould  have  the  fol- 
lowing features: 

1)  The  computer  v/as  to  allow  efficient  image  processing  at  high 
speed  utilizing  interactive  computation,  making  possible 
large  data  evaluation. 

2)  The  computer  v/as  to  preserve  the  general  purpose  aspects  of  a 
general  purpose  corrpu;er  for  image  processing. 

3)  The  computer  v/as  to  be  cost  effective  in  order  to  allow  In- 


dustrial realization. 
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The  framework  of  this  proposed  computer  architecture  consists  of  a 
task  management  processor,  a parallel  processor,  and  a sequential  arith- 
metic processor.  The  task  management  processor  Is  a set  of  software 
system  programs  serving  as  an  operating  system.  The  parallel  processor 
is  proposed  because  of  the  parallel  nature  of  the  operations  involved 
In  image  processing.  Parallel  machines  are  considered  to  be  particularly 
suitable  for  image  processing  by  Thurber  and  Wald  [102].  The  parallel 
processor  of  the  proposed  computer  architecture  consists  of  an  array  of 
microprocessors  which  allow  parallel  processing  capability..  This  parallel 
processor  Is  a homogeneous  system  in  which  all  processors  are  alike  and 
are  general  purpose  units.  The  homogeneous  parallel  processor  lends 
itself  quite  readily  to  extendabi 1 i ty  as  such  systems  are  usually  modular- 
ly  constructed.  With  modular  parallel  processing,  the  system's  memory, 
processor,  and  input/output  nodules  may  be  enlarged  as  processing  re- 
quirementi.  increase;  thereby  avoiding  replaces- .at  of  the  entire  parallel 
processing  system.  The  modular  parallel  processor  also  provides  very 
high  reliability  since,  with  several  identical  modules  of  each  types, 
the  system  can  withstand  failures  in  several  modules  acd  still  operate. 
This  arrangement  also  increases  efficiency  and  through-put  since  all  of 
the  processors  could  be  operating  simultaneously.  The  sequential  arith- 
metic processor  is  a microprogrammed  controlled  processor  which  performs 
the  sequential  arithmetics  and  also  controls  the  Input/output  devices. 

The  details  of  these  processors  will  be  set  forth  in  section  5.2.  it 
will  be  shown  that  the  design  goals  were  achieved  through  the  proposed 
computer  architecture. 
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5.1  PREVIOUS  WORK  AND  COMMENTS 

As  mentioned  in  Chapter  1,  previous  wor*k  in  the  field  of  special 
purpose  computer  architecture  for  image  processing  basically  falls  into 
two  categories:  bit-plane  processing  and  distributed  processing.  The 
bit-plane  processing  approach  performs  the  arithmetic  computation  on  the 
Image  points  which  are  stored  in  Boolean  bit-plane.  For  example,  if  the 
image  has  eight  grey  levels,  then  the  image  points  are  stored  in  three 
Boolean  planes.  The  bit-plane  processing  approach  tends  to  have  a large 
number  of  processors  which  perform  Boolean  operations.  The  distributed 
computing  approach  utilizes  processors  which  have  powerful  computation 
capability.  These  processors  are  designated  by  microprogram  or  hardware 
to  execute  certain  specific  tasks.  This  configuration  forms  a 
distributed  computing  architecture.  In  this  section  we  will  briefly 
illustrate  each  special  purpose  computer,  concentrating  on  the  special 
features  of  each  computer  for  image  processing. 

The  Illinois  Pattern  Recognition  Computer,  I LL 1 AC  III,  [87,100]  v/as 
designed  for  automatic  scanning  and  analysis  of  massive  amounts  of 
relatively  homogeneous  visual  data,  in  particular,  bubble  chamber 
negatives.  The  computation  is  performed  by  three  units,  the  pattern 
articulation  unit,  the  taxicrinic  unit,  and  the  arithmetic  unit.  The 
pattern  articulation  unit  performs  local  preprocessing  on  the  digitized 
raster,  such  as  track  thie-iing,  gap  filling,  line  element  recognition, 
etc.  The  logic  desing  of  the  digital  computer  has  been  otpimized  for 
the  idealization  of  the  input  image  to  a line  drawing.  Nodes  represent- 
ing end  points,  bends,  points  of  intersection  are  labeled  in  parallel  by 
appropriate  programs.  The  abstract  graph  describing  the  Interconnection 
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of  labeled  nodes  Is  then  extracted  as  a list  structure,  which  comprises 
the  normal  output  of  the  processing  element  (stalactite).  The  pattern 
articulation  unit  consists  of  an  Iterative  array  of  1024  identical  sta- 
lactites (32x32).  A schematic  diagram  of  the  stalactite  is  shown  In 
Figure  5.1.  The  iterative  array  of  the  stalactite  can  be  connected  In, 
either  rectangularly  or  hexagonally,  at  the  programmer's  option.  Each 
stalactite  can  accept  an  input  from  any  of  the  eight  neighboring  sta- 
lactites, SO,  Si,  ...»  and  S7  shown  in  Figure  5.1  (rectangular)  or  six 
neighboring  stalactites  (hexagonal)  in  the  plane  and  from  Itself,  M 
shown  in  Figure  5.1.  The  input  signals  are  logically  "0R,,ed,  optionally 
complemented,  and  stored  in  one  (or  more)  of  the  nine  planes.  Communi- 
cation with  the  supplementary  planes  in  the  core  buffer  is  through  the 
"M"  plane,  which  serves  as  the  buffer  register  of  this  memory.  For 
output,  the  output  of  any  selected  set  of  planes  can  be  logically  "AND"ed 
and  passed  on  to  neighboring  stalactites.  With  a special  signal,  the 
stalactite  allows  an  input  signal  to  pass  through  it  directly,  without 
interim  storage.  It  is  this  feature  which  allows  path-building  within 
the  machine.  The  set  of  32x32  stalactites  processes  the  pixels  of  a 
32x32  pixel  image  windo w simultaneously.  The  whole  image  is  to  be 
processed  by  one  image  window  (32x32)  after  another  sequentially  by  the 
set  of  32x32  stalactites.  The  taxicnnlc  unit  assembles  the  graphs, 
which  are  outputs  of  the  pattern  articulation  unit.  Into  coherent  list 
structures  and  categorizes  these  graphs  to  complete  the  visual  recogni- 
tion function.  The  arithmetic  unit  is  designed  for  executing  mathematical 
analysis,  such  as  stereoreconstruction,  statistical  summarization,  etc. 
involved  in  processing  pictorial  data.  The  block  diagram  of  1LLIAC  III 
Is  Illustrated  In  Figure  5.2. 
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Figure  5.2.  Block  diagram  of  Illinois  pattern 
recognition  computer  ILLIAC  III. 
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One  special  feature  of  I LL l AC  ill  for  image  processing  is  the  use  of 
an  auxiliary  memory.  The  auxiliary  memory  is  used  for  the  processing  of 
images  frame  to  frame,  and  for  retaining  intermediate  partial  results. 

The  necessity  of  storing  intermediate  results  of  the  Iterative  array 
stems  directly  from  the  manner  of  executing  homogeneous  logic  transforma- 
tions ?n  the  processor.  The  complexity  of  the  iterative  array  can  be 
greatly  simplified  if  various  digital  filtering  operations  can  be  per- 
formed serially  storing  the  intermediate  results  (32x32  = I02*f  bit  words) - 
For  example,  local  track  segment  orientation  can  be  designated  as  being 
predominantly  horizontal,  vertical,  left-diagonal,  or  right-diagonal  In 
four  serial  tests,  whereas  simultaneous  identification  requires  approxi- 
mately four  times  as  much  hardware  [87].  Unfortunately,  ILLIAC  III  has 
never  been  completely  built. 

The  use  of  an  auxiliary  memory  for  Image  processing  has  been  imple- 
mented in  several  computer  facilities  for  pattern  recognition  research. 

One  example  is  the  Purdue  University  Advanced  Automation  Research  Labora- 
tory [101],  This  laboratory  is  organized  around  a DEC  PDP  11 A5  digital 
computer  with  32K  core  memory  and  96  K fast  secondary  memory  [1073,  two 
disk  drives,  a magnetic  tape  drive,  two  cassette  tape  drives,  a line 
printer,  and  a CRT  monitor.  The  system  block  diagram  is  shown  in 
Figure  5.3.  The  computer  is  not  a special  purpose  computer,  but  the 
auxiliary  memory  is  a special  feature  for  image  processing.  The  auxiliary 
memory  was  developed  for  Image  processing  ~;his  laboratory,  mainly  be- 
cause of  the  limited  addressing  range  of  the  16-bit  minicomputer. 
Therefore,  the  limited  addressable  memory  is  difficult  to  use  for  im- 
plementing large  programs.  A memory  controller  has  been  built  which  can 
access  up  to  2 bytes  of  memory,  which  now  controls  6hl\  16-bit  words 
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Figure  5.3.  Block  diagram  of  computer  system  in  Advanced 
Automation  Research  Laboratory. 


of  core  memory  [107].  The  controller  is  interfaced  to  the  POP  11/45 
unibus  through  using  three  56-bit  registers.  Two  registers  for/n  an 

effective  address  of  32  bits,  and  the  third  register  contains  the  data 
to  be  written  into  the  memory  or  the  data  read  from  memory.  Once  the 
address  ?s  loaded  into  the  two  registers,  data  can  be  written  into  or  read 
from  memory  in  less  than  one  microsecond.  Since  large  arrays  of  data 
can  be  addressed  without  1/0  ro  the  disks,  execution  times  improve. 

Also,  computer  programs  become  simpler  since  no  sophisticated  disk-core 
swapping  software  is  needed.  An  experiment  on  rib  identification  in 
chest  x-ray  images  v/as  programed  once  without  using  the  auxiliary  memory 
and  once  using  this  memory.  On  the  average,  the  execution  time  improved 
a factor  of  five.  Other  software  shows  comparable  execution  time  [101 ]. 

Digital  Parallel  Processors  (DPP's)  [89,90,91],  using  cellular  logic, 
are  real-time  machines  in  which  the  action  resulting  from  a program 
statement  is  simultaneous  on  all  the  points  of  the  array.  The  action 
may  be  symmetrical  or  directional  and  the  tesselations  of  various  types, 
the  most  cor.inon  being  the  square  and  the  hexagonal  ones.  An  example  nf 
this  kind  of  machine  is  the  parallel  cellular  logic  image  processor, 

CLIP3  [91].  The  CL1P3  is  comprised  of  an  array  of  132  cells  arranged  in 
a block  of  dimensions  16  cells  (vertically)  by  12  cells  (horizontally). 
The  array  interconnection  pattern  can  be  either  square  or  hexagonal.  The 
required  architecture  is  determined  by  one  bit  in  each  instruction  word 
and  is  therefore  under  the  control  of  the  programmer.  The  block  diagram 
of  the  cell  logic  of  CLIP3  is  shown  in  Figure  5.4.  The  Boolean  processor 
can  perform,  under  program  control,  two  independent  Boolean  functions 
from  its  two  inputs  A and  P.  D is  one  output  and  can  be  regarded  as  the 
processed  pattern  bit  corresponding  to  the  cell.  The  other  output,  II, 
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fans  out  to  the  adjacent  cells.  The  N Inputs  from  neighboring  cells  (G  , 

G2,  and  Gg)  are  summed  (e)  and  compared  v/lth  a threshold  (f ) . The 

programmer  Is  allowed,  for  each  PROCESS  instruction,  to  select  several 
Inputs  to  be  put  into  the  summing  unit  and  he  can  also  choose  the  thres- 
hold value.  The  summed  and  thresholded  output,  T,  is  then  ,,0R,,ed  with 
the  second  pattern  input  to  form  the  input,  P,  to  the  Boolean  processor. 

D outputs  are  addressed  into  ny  one  of  14  single  bits  of  storej  buffers 
A and  B are  loaded  from  addressed  location  in  the  same  store.  The  CLIP3 
was  actual  ly  built. 

One  drawback  of  both  the  ILLIAC  ill  and  CL1P3  is  that  the  "edges" 
caused  by  moving  image  windows,  (in  ILUAC  Hi  32x32  windows  and  16x12 
window  in  CUP3)  are  not  taken  care  of  by  the  computer.  The  computing 
power  of  the  CLIP3  Is  obviously  limited  because  cf  the  fixed  small 
number  of  (Boolean  operator)  cells.  (16x12  * 192  cells) 

In  order  to  process  larger  Images,  CLIP3  has  been  Interfaced  to  a 
television  camera  through  the  hardv/ired  scanning  unit.  This  Is  called  a 
hybrid  CLIP3  array  [92]  which  is  shown  schematically  In  Figure  5.5.  This 
unit  scans  the  192  cell  CLIP3  array  across  the  96  by  96  cell  data-field 
and  provides  edge  stores  to  handle  the  propagation  signals v.'hich  cross 
between  adjacent  sectors.  The  complete  system  is  interfaced  to  a PDP 
11/10  computer  which  serves  to  extend  the  available  data  and  instruction 
storage  and  also  provides  program  editing  and  assembling  facilities. 

CI.IP4  [92]  is  the  Large  Scale  integrated  circuit  (LSI)  version  of 
CL1P3  with  some  small  changes  in  the  cell  design.  The  CLIP*!  uses  H-MOS 
(i:  type  Metal  Oxide  Silicon)  LSI  to  incorporate  eigh^  processor  cells  In 
a four  !>y  two  block  onto  one:  chip.  The  block  diagram  of  CLIP4  cell  logic 
Is  shown  in  Figure  5.6.  The  "D"  store  has  been  Increased  from  the  8 bits 
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5.5.  Schematlcal  diagram  of  hybrid  CLIP  3 system. 
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of  CUP3  to  32  bits  which  allow  the  processing  of  32  grey  levels.  The 
Interconnecting  threshold  gate  has  been  replaced  by  an  ,,0R"  gate.  •-  A'  - 
few  extra  gates  and  an  additional  buffer  store  have  been  included  to 
provide  automatic  carry-in  arithmetic  operations. 

Comparing  the  performance  of  CLIP3  and  CLIP4,  the  processing  speed 
of  the  CLIP**  cell  is  slower  than  that  of  CLIP3  by  at  least  a factor  of 
five.  A single  operation  which  required  a pair  of  instructions  (LOAD 
and  PROCESS)  in  CLIP3,  taking  2 psec,  is  expected  to  require  about  10  ps 
in  CLIP**.  Propagation  from  cell  to  cell  took  0.1  ps  in  CLIP3  and  may 
take  1 ps  in  CLIP*!  [92].  However,  the  processing  speed  of  the  overall 
system  of  the  CLIPA  is  improved  by  the  larger  array  of  processors  of 
CL  1 pit  (96x96  cells).  The  CLIP**  has  not  been  completely  built  up  to  now. 

The  Parallel  P’cture  Processing  Machine  (PPM)  [C8]  is  a special 
processor  which  is  connected  to  and  controlled  by  a conventional  computer. 
The  block  diagram  of  PPM  is  shown  in  Figure  5.7(a).  The  PPM  consists  of 
the  following  principal  parts:  the  processing  unit,  a set  of  nine 
general  purpose  picture  registers,  and  a control  unit.  The  picture 
registers  which  are  shown  in  Figure  5.7(b)  comprise  nine  shift  registers 
capable  of  storing  a picture.  The  main  parts  of  the  processing  unit  are 
the  neighborhood  matching  logic  (NHL),  shown  in  Figure  5.7(c),  the  line 


Figure  5.7(a).  Block  diagram  of  parallel  picture  processing  machine 
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Figure  5.7(b).  Picture  registers. 
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conventional  In  all  respects  except  for  the  writable  memory  where  the 
templates  of  an  operation  to  be  performed  are  stored.  The  parallel  pic- 
ture processor  in  [83]  has  been  linked  with  an  ordinary  minicomputer  as 
part  of  the  computation  facilities  of  the  PICAP  picture  processing  labora- 
tory at  the  University  of  Llnkoplng,  Sweden.  The  block  diagram  of  the 
system  is  shown  In  Figure  5.7(d).  The  function  of  the  minicomputer  is 
twofold:  (l)  it  controls  the  other  devices  and  (2)  it  performs  all 
nonpictorial  data  processing.  PICAP  which  is  a modified  version  of  PPM 
is  dedicated  to  the  several  tasks  of  the  picture  processing,  such  as 
fingerprint  coding  and  malaria  parasite  detection.  An  important  modifi- 
cation of  PICAP  has  been  the  addition  of  a set  of  registers  for  the 
collection  of  measurements.  The  most  important  task  of  the  picture  pro- 
cessor is  to  produce  measurements  of  application-dependent  features. 

That  is,  to  reduce  the  often  enormous  amount  of  information  in  an  image 
to  a set  of  feature  measurements  that  can  be  handled  by  a conventional 
computer.  The  syntactic  method  for  fingerprint  classification  can  only 
be  handled  by  the  conventional  computer  In  the  PICAP  computer  of  Kruse 
[99].  The  fact  that  they  cannot  perform  syntactic  analysis  is  a drawback 
cf  the  PPH's  (or  PICAP's). 

Using  the  approach  of  distributed  computation  for  designing  a 
special  purpose  computer  for  image  processing,  the  Control  Data  Corpora- 
tion designed  the  Flexible  Processor  [97].  The  Flexible  Processor  was 
developed  for  a large  digital  change  detection  system  for  concurrent 
processing  of  four  channels  of  side-looking  radar  Imagery.  The  Flexible 
Processor  is  a microprogrammable  processor  and  uses  random  access  memory 
up  to  1024  words  of  48  bits  each  for  microprogram  control.  One  special 
feature  of  the  Flexible  Processor  is  its  data  transmission  which  Is 


shown  in  Figure  5.8.  Four  separate  balanced  party-line  transmission 
channels  are  available.  Each  32— b i t channel  can  Initiate  and  receive 
register-buffered  transfers.  A scanning  system  allows  several  Flexible 
Processors  on  a party-line  to  communicate  with  each  other  and  with  peri- 
pheral devices.  For  output  transmission,  a 32-bit  output  channel  is 
provided  which  interfaces  on  balanced  lines.  A dual  internal  data  bus 
system  is  used  to  match  data  transfer  speed  to  the  speed  of  arithmetic 
logic.  The  arithmetic  unit  consists  of  an  arithmetic  logic  unit,  hard- 
ware network  for  conditional  microinstruction  execution,  array  hardware 
multiplier,  specialized  logic  for  square  root  and  divide,  and  hardware 
priority  interrupt  mechanism. 

Another  special  computer  for  Image  processing,  utilizing  the  distri- 
buted computing  approach,  was  developed  by  Toshiba  Company  called  Toshiba 
image  Processing  System  TOSPICS  [93,116].  The  TOSPICS  in  an  interactive 
image  processing  system  which  is  a disk-based  system  and  each  operation 
is  performed  by  the  command  input  through  a teletypewriter.  The  software 
system  for  the  image  processing  system  consists  of  the  permanent  and  non- 
permanent resident  system  programs,  the  picture  processors,  the  block 
common  area,  and  a package  of  image  processing  programs.  The  system 
diagram  of  TOSPICS  is  shown  in  Figure  5*9.  The  special  features  of 
TOSPICS  for  image  processing  are  that  the  image  memory  is  commonly  used 
in  order  to  reduce  the  amount  of  data  transmission  and  the  parallel 
picture  processor  is  constructed  to  perforn  certain  programs  at  high 
speed.  The  parallel  picture  processor  performs  the  program  of  spatial 
filtering  at  a speed  of  1 pixel/i  psec.  The  I/O  devices  of  TOSPICS  in- 
clude a unique  high  precision  flying  spot  scanner  by  a Double  Deflection 


Tube  (DDT)  with  hQ^xkOSS  resolvable  points  and  32,000x32,000  addressable 
points. 

Comparing  the  systems  of  Flexible  Processors  and  T0SPICS,  the  basic 
Ideas  of  parall;  ■ firocessing  for  Image  processing  by  the  distributed 
computing  approach  are  the  same  for  these  tv/o  systems.  The  difference 
between  these  two  computer  systems  Is  the  ways  of  utilizing  processing 
elements.  The  T0SPICS  uses  the  minicomputer,  TOSBAC-ffOC,  as  an  Image 
processor  and  a parallel  picture  processor  is  attached  to  it.  The 
s iral’el  picture  processor  used  for  certain  programs,  such  as  spatial 
filtering.,  afin-c.  transformations,  histogram  calculation,  and  density 
conversio  *-  :.li3) . The  computer  system  of  Flexible  Processors  uses  the 
Flexible  Processors  as  the  processing  elements.  Each  Flexible  Processor 
is  assigned  for  one  task  and  several  Flexible  Processors  are  organized  as 
the  computer  system. 

The  STARAN  [108]  computer  is  a parallel  processing  computer  built 
by  Goodyear  Aerospace  Company  In  1972.  STARAN  was  not  designed  as  a 
special  prupose  computer  for  Image  processing.  But  this  computer  was 
utilized  for  the  task  of  resampling  in  the  field  of  Image  processing  in 
1977  [109].  The  STARAN  parallel  processor  is  a single  Instruction  stream 
multiple  data  stream  (SIKD)  processor.  The  previously  reviewed  computers, 
such  as,  1LUAC  II!,  CLIPA,  PICAP,  and  T0SPICS  are  SIMD  processor.  The 
Flexible  Processor  Is  SISD  processor.  The  single  most  Important  element 
of  STARAN  Is  the  associative  array,  which  provides  content  addressability 
and  parallel  Instruction  execution  capabilities.  Most  STARAN  computing 
Is  done  within  a word  of  associative  array  memory.  An  associative  array 
word  is  normally  divided  into  fields  of  varying  lengths  by  the  programmer 
to  suit  the  requirements  of  specific  programs.  The  values  of  these 


*{  **•  #w  >w*>5 1 


188 


flel  is  then  can  be  added,  subtracted,  multiplied,  and  divided  within  the 
word.  The  STARAN  can  perform  the  same  operations  as  a seauential  processor 
but  with  the  added  capability  of  performing  these  operations  simultaneous- 
ly on  literally  thousands  of  words  in  the  associative  processor  arrays. 

A Lcjic  STARAN  configuration  contains  an  associative  array.  However,  up 
to  32  associative  arrays  can  be  include 1 in  a single  STARAN  system.  The 
block  diagram  of  the  STARAN  computer  s shown  in  Figure  5.10.  The 
sequential  controller  provides  off-line  capabilities  for  assembling  and 
debugging  STARAN  programs  and  cos  . ••'il  of  STARAN  error  processing, 
diagnostic,  and  maintenance  programs.  Buffered  I/O  is  available  for 
tying  different  types  of  peripherals  into  the  STARAN  control  memory. 

Also  BI/0  can  be  used  to  transfer  blocks  of  data  and/or  programs  between 
the  STARAN  control  memory  and  host  memory.  The  external  function  (EXF) 
logic  facilitates  coordination  between  the  different  elements  of  STARAN 
for  special  functions  and  simplifies  housekeeping,  maintenance,  and 
test  functions.  By  issuing  external  function  codes  to  the  EXF  logic, 
elements  of  STARAN  can  control  and  interrogate  the  status  of  other 
elements.  In  general,  the  STARAN  computer  is  powerful  compared  with  the 
Flexible  Processor  and  TOSPICS  However,  only  several  image  processing 
tasks  (such  as  resampling)  have  been  carried  out  on  this  computer. 
Therefore,  the  effective  utilization  of  STAR.JI  for  all  types  of  image 
processing  is  still  an  open  question. 

Consider  the  attributes  of  the  bit-plane  processing  approach  and 
the  distributed  computing  approach.  The  bit-plane  approach  mostly  uses 
Boolean  operators  as  processors.  Boolean  operators  work  only  on  binary 
images  which  are  not  common  In  the  real  v/orld.  One  way  to  get  around 
this  is  to  use  several  binary  picture  planes  to  represent  the  grey  scale 
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values  of  picture  points.  But,  the  complex  software  and  additional 
memory  requirement  cause  another  problem  and  this  problem  limits  the 
processing  power  of  the  processor.  One  of  the  drawbacks  of  the  bit-plane 
processing  approach  is  that  the  processing  power  of  the  processor  may 
not  be  adequate  for  some  of  the  mere  sophisticated  image  processing 
techniques.  For  example,  Kruse  [993  has  shown  that  the  parallel  Picture 
Processing  Machine  PPM  (now  called  P1CAP)  [88]  to  be  applicable  only  to 
the  preprocessing  part  of  fingerprint  classification  and  as  stated  above 
syntactic  techniques  have  to  be  performed  by  the  conventional  computer 
in  the  PICAP  system.  Thus,  the  ability  of  the  bit-plane  approach  to 
perform  highly  sophisticated,  but  important,  techniques  is,  at  this  time, 
unsure.  The  capability  of  real-time  processing  of  the  bit-plane  pro- 
cessing computers  such  as  CLIP3,  CLIPll,  and  PPM  will  be  very  difficult 
to  ascertain  until  more  complicated  techniques  applied  to  real  world 
pictures  v/ich  more  grey  levels  such  as  128  or  258,  have  been  implemented 
by  these  computers. 

After  studying  the  feasibilities  of  the  bit-plane  processing  and 
distributed  computing  approaches  to  the  real  world  image  processing 
task,  v/e  feel  that  the  distributed  computing  approach  is  better 
considering  the  present  state-of-the-art  with  respect  to  both  software 
and  hardware,  in  March  1977,  Stone  [106]  indicated  that  the  distributed 
computing  approach  is  one  of  the  future  trends  for  general  computer 
architecture.  His  remark  supports  our  judgment  on  the  distributed  com- 
puting approach  for  special  computer  architecture  for  image  processing. 

A major  drawback  of  previous  computers  designed  by  distributed  computing 
approach  is  that  the  system's  processors  are  not  reconfigurable.  The 
vast  varieties  of  sensor  types,  applications,  and  image  processing 
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techniques,  require  that  the  image  processing  system  (especially  the 
parallel  processor)  be  reconf Igurable.  Therefore,  a generalized  computer 
architecture  which  is  reconf igurable  under  software  control  is  proposed 
in  the  next  section  of  this  chapter  for  the  many  applications  of  image 
processing.  Not  only  is  the  concept  of  reconf igurable  ability  new  as 
special  purpose  computer  architecture  for  image  processing,  but  also 
the  methods  of  exploitation  of  parallelism  is  new.  In  the  proposed 
computer  architecture  parallelism  within  the  task  is  exploited  by  the 
parallel  processor.  In  the  meantime  the  operations  of  the  sequential 
arithmetic  processor  are  pipelined  with  the  parallel  processor  under 
programmer  control  in  certain  tasks  which  can  be  decomposed  into  pipe- 
lined processing.  Therefore,  parallelism  and  pipelining  are  exploited 
at  the  same  time  in  the  proposed  computer  architecture.  The  parallelism 
[104]  used  is  the  multiprocessing  approach  which  subdivides  each  out- 
coming  job  among  many  identically  constructed  mechanisms.  The  pipe- 
lining, or  overlap  [105]  processing  is  another  multiprocessing  approach 
which  ?s  to  develop  a collection  of  specialized  mechanisms  capable  of 
working  simultaneously  to  form  a general  purpose  organization.  The 
processing  time  of  the  image  processing  task  by  the  proposed  computer 
architecture  will  be  sped  up  by  a factor  which  is  comparable  to  the 
amount  of  parallelism  and  pipelining  existing  in  the  image  processing 
task  of  interest. 

5.2  PHYSICAL  ORGAN  I ZAT I ON  AND  CONTROL  FLOW  OF  THE 
PRQ?}QSED  COMPUTE!!  ARCHITECTURE 

The  proposed  Computer  Architecture  for  jmage  Processing  Is  called 

CAIP  and  Is  to  be  designed  using  the  most  recent  semiconductor  technology. 
The  physical  organization  and  control  flow  are  described  in  next  sub- 
sections. 
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5.2.1  Physical  Organization  of  the  Proposed  Computer  Architecture 

The  physical  organization  of  this  special  computer  architecture  for 

image  processing  (CAIP)  comprises  the  task  management  processor,  the 

I 

control  units,  the  parallel  processor,  the  sequential  arithmetic  pro- 
cessor, and  the  memory  organization. 

1)  Task  Management  Processor  (TMP)  Is  a set  of  software  programs 
which  allocate  the  jobs  to  the  parallel  processor  (PP)  or  Sequential 
Arithmetic  Processor  (SAP).  The  set  of  software  programs  include  a 
task  control  program,  a job  control  program,  an  input/output  program, 
and  the  language  translation  program.  The  task  control  program  provides 
the  logical  interface  between  the  hardware  and  the  remainder  of  the  soft- 
ware system  and  is  responsible  for  the  allocation  of  jobs  to  the  parallel 
processor  and  sequential  arithmetic  processor.  Each  task  has  a tag  which 
is  designated  by  the  programmer  for  the  identification  of  parallel  pro- 
cessing or  sequential  processing.  One  part  of  the  task  control  program 
Is  called  the  tag  examination  program  which  examines  the  tags  on  tasks 
and  allocates  the  tasks  to  the  proper  processor.  Following  the  tag 
examination,  the  initiation  program,  which  is  another  part  of  the  task 
control  program,  initiates  the  parallel  processor  or  the  sequential 
arithmetic  processor.  In  general,  the  task  control  program  performs 
scheduling,  supervision,  interruption  handling,  execution  supervision, 
and  clock  supervision.  The  job  control  program  provides  a logical  inter- 
face between  a task  and  a job  or  between  a task  and  the  system  operator. 
The  job  control  program  analyzes  the  job  stream,  looks  at  system  re- 
sources, processes  job  execution  and  termination,  and  communicates 
between  the  system  operator  and  the  individual  job  program.  The  I/O 
control  program  provides  an  Interface  between  the  processing  programs 
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and  the  I/O  devices.  The  I/O  control  program  performs  I/O  supervision, 
access  rout.ne  processing,  and  I/O  device  initiation.  The  language 
translator  program  translates  the  computer  language  into  machine  codes 
and  compiles  the  program  to  be  executed. 

2)  The  Control  Units  (CU's)  consist  of  two  sets  of  software  pro- 
grams. One  control  unit  (CUPP)  Is  for  the  parallel  ,:ocessor,  and  the 
other  (CUSA)  is  for  the  sequential  arithmetic  processor.  The  CUPP  and 
CUSA  are  different  from  conventional  processors,  in  that  they  are  soft- 
ware programs  which  control  the  operation  of  the  parallel  processor  and 
the  sequential  arithmetic  processor  respectively.  The  CUPP  is  a control 
program  which  initiates  two  different  sets  of  software  operating  systems, 
one  for  51MD  mode  and  the  other  for  MIMD  mode.  The  two  sets  of  software 
operating  systems  drive  the  parallel  processor  Individually  upon  the 
command  of  CUPP.  The  reconfiguration  from  SIMD  mode  to  MIMD  mode  or 
MIMD  mode  to  SIMD  mode  Is  performed  by  loading  the  operating  system 
corresponding  to  the  desired  mode.  Ilext,  the  operating  system  is 
assigned  to  the  parallel  processor  (PP)  by  the  control  program  of  CUPP. 
Hence,  the  parallel  processor  operates  in  either  SIMD  or  MIMD  modes 
under  the  respective  operating  systems.  Through  this  arrangement,  the 
CUPP  reconfigurates  the  computer  architecture  from  SIMD  to  MIMD  or  MIMD 
to  SIMD.  This  reconf igurable  capability  enables  this  computer  architec- 
ture to  satisfy  the  large  variety  of  applications  of  image  processing. 

The  operating  system  of  MIMD  mode  includes  scheduling  routines,  dynamic 
allocating  routines,  and  dispatching  routines.  The  scheduling  routines 
schedule  each  job  depending  on  job  priority  and  facility  requirements. 
The  dynamic  allocating  routines  take  jobs  set  up  by  the  scheduling  rou- 
tines and  partition  the  set  of  processors  according  to  the  need  of  each 
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job.  The  dispatching  routine  dispatches  the  processors  when  the  job 
terminates  or  some  higher  priority  task  requires  processors.  The  operat- 
ing system  for  the  SiMD  mode  is  on  the  master  control  unit.  All  the 
processors  of  the  parallel  processor  (PP)  are  controlled  by  this  master 
control  unit  and  thereby  an  instruction  is  executed  simultaneously  on 
all  the  processors.  The  control  unit  of  the  sequential  arithmetic 
processor  (CUSA)  controls  the  sequential  arithmetic  processor  which  is 
a microprogram-controlled  bipolar  processor.  The  control  units  are 
shown  in  Figure  5.11.  llote  tiiat  Figures  5.11 » 5.12,  5.13,  and  5.14  form 
a graphical  illustration  of  this  computer  architecture  (CAIP)  by  linking 
the  corresponding  symbols  a,  fl,  y,  and  a in  the  figures. 

3)  The  Parallel  Processor  (PP)  is  on  array  of  microprocessors.  For 
the  parallel  processor,  N microprocessors  are  connected  in  an  array 
fashion.  The  array  organization  is  suitable  for  Image  processing 
[102],  The  number  N is  determined  from  the  tradeoff  considera- 
tions between  performance  and  cost.  The  optimal  number,  H,  varies  with 
the  task.  Hence,  H can  only  be  determined  at  the  time  of  implementation. 

In  the  framework  shown  in  Figure  5.12,  a set  of  64  microprocessors  is 
used  to  give  an  idea  of  the  dimension  of  the  problem.  The  control  unit 
(CUPP)  controls  this  set  of  microprocessors  in  SiMD  or  MIMD  modes.  This 
control  unit  enables  the  parallel  processor  (PP)  to  have  a higher  degree 
of  flexibility  and  processing  power.  The  SIMD  mode  utilizes  a single 
master  control  unit  which  drives  the  multiple  processing  units  (micro- 
processors), all  of  which  either  execute  or  ignore  the  current  instruc- 
tion. This  SIMD  mode  is  especially  useful  for  the  cases  in  which  there 
exists  (1)  a large  amount  of  Independent  data,  (2)  no  restrictions 
preventing  them  from  being  processed  in  parallel,  (3)  a requirement  for  j 
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PROPOSED  ARCHITECTURE  OF  SPECIAL  PURPOSE  ARRAY  PROCESSING, 
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Figure  5.11.  Control  units  of  designed  computer  architecture 
for  image  processing. 
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high  throughput,  and  (h)  a possibility  of  exploiting  the  associative 

i 

addressing  selection  technique.  Thus,  SIMD  mode  is  suitable  for  the 

local  task,  which  executes  the  same  instruction  on  each  picture  element 

✓ 

v/ithin  an  image  window.  The  HIMD  mode  utilizes  N processors  and  II 
memories  where  each  processor  follows  an  independent  instruction  stream. 
The  parallel  processor  (PP)  is  connected  by  crossbar  switches  to  an 
interleaved  memory  system  which  divides  the  ordinary  memory  into  modules 
and  the  consecutive  data  are  stored  in  different  nodules.  The  inter- 
leaved memory  system  is  used  because  the  bandwidth  is  greater  than  a 
conventional  memory  system  . The  Interleaved  memory  system  is  more 
appropriate  for  parallel  processing  than  a conventional  system,  in 

which  data  can  only  be  accessed  one  at  a time  [9*1  ].  The  cost  of  cross- 
bar switches  is  high,  since  every  processor  can  communicate  with  the 
other  processors  and  memory  modules.  Depending  on  the  inage  processing 
{ tasks  specified  by  the  user,  the  interconnection  among  processors  and 

memory  nodules  needed  for  these  tasks  can  be  studied,  the  alternatives 
for  the  crossbar  switches  might  be  selected. 

Ij)  The  Sequential  Arithmetic  Processor  (SAP)  is  a microprogram- 
controlled  processor.  Mini  and  micro- computers  are  not  used  here 
because  user  microprogrammable  capability  and  bipolar  processor  are  not 
furnished  by  usual  mini  or  micro-computers.  The  Sequential  Arithmetic 
Processor  (SAP)  Is  a bipolar  processor  which  is  a processor  built  by 
bipolar  semiconductor  technology  and  usually  has  the  bit  slicing 
capability.  The  bipolar  microprogrammable  processor  permits  the  de- 
signer to  define  his  own  instruction  set  and  the  associated  hardware 
architecture  to  achieve  special  capabilities,  such  as,  variable  word 
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length  capability  or  to  perform  an  application  with  the  highest 
efficiency.  The  bipolar  processor  expands  the  CPU  word  length  by  cas- 
cading the  needed  number  of  bit-slice  microprocessor  components.  This 
variable  word  length  capability  of  SAP  makes  It  more  general  and  power- 
ful than  other  microprocessors.  Along  with  this  processor,  a mlrro- 
program  memory  is  needed  to  store  the  microprograms  of  certain  programs 
frequently  In  use.  For  the  microprograms  of  Image  processing  tech- 
niques, no  Instruction  fetching  is  required  because  of  the  coding  of 
the  microprogram.  The  microprogram  capability  saves  processing  time 
according  to  the  ratio  of  Instruction  fetch  time  to  total  execution 
time.  The  Sequential  Arithmetic  Processor  Is  shown  In  Figure  5.13. 

5)  The  Memory  Hierarchy  Organization  Is  the  memory  system  for  SAP. 
The  picture  is  stored  on  a magnetic  disk  memory  which  Is  more  economical 
than  core  memory,  but  memory  access  time  Is  long.  The  memory  access  time 
of  the  bipolar  memory  is  faster  by  a factor  of  100  than  disk  memory  [111]. 
Therefore,  a semiconductor  bipolar  memory  is  connected  to  the  disk  memory 
as  working  mcnory  space  and  buffer.  The  hierarchy  organization  is  as 
follows:  the  picture  area  which  is  to  be  processed  is  loaded  onto  the 
bipolar  memory  from  the  disk  memory,  then  the  processor  gets  the  data 
(picture  points)  from  the  bipolar  memory  thus  allowing  extremely  fast 
memory  access  time.  In  order  to  avoid  being  delayed  by  the  loading 
time  from  disk  to  bipolar  memory,  a Bipolar  Memory  Buffer  (BMB) , is 
used.  V/hile  the  processor  is  reading  the  data  Into  the  bipolar  working 
memory  sapce,  the  next  picture  area  is  loaded  on  the  Bipolar  Memory 
Buffer  (BMB).  Thus,  this  memory  preloading  makes  the  data  always  ready 
In  the  fast-acccss  bipolar  memory.  This  memory  hierarchy  organization 


Figure  5.13*  Sequential  arithmetic  processor  of 
designed  computer  architecture. 
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Js  illustrated  in  Figure  5.14.  This  memory  hierarchy  organization  is 
different  from  the  memory  of  TOSPICS  [116]  and  the  inage  core  memory  of 
the  11/45  [1073.  The  approach  of  the  design  of  the  memory  system  of 
TOSPICS  [116]  was  to  employ  a large  memory  storage  to  reduce  the  memory 
loading  time.  The  approach  in  designing  the  secondary  core  memory  of 
the  PDP  11/45  [1073  used  primarily  for  image  storage,  was  to  utilize 
this  secondary  storage  as  a buffer  memory  as  in  the  auxiliary  memory 

concept  of  the  I LL I AC  IN  [1003  . However,  the  secondary  core  memory  of 
the  PDP  11/45  is  not  randomly  accessible.  Each  time  an  Image  datum  is 
needed  the  user's  program  has  to  request  that  datum  to  be  loaded  to  the 
image  core  memory.  The  proposed  memory  hierarchy  loads  a large  block  of 
data  onto  the  Dipolar  Buffer  Memory  as  image  processing  necessitates 
operations  on  large  blocks  of  data,  this  ability  of  the  CAIP  through  its 
BMD  provides  a marked  advantage  for  image  processing.  Once  the  large 
block  of  data  has  been  loaded  onto  the  bipolar  memory,  any  individual 
data  point  can  be  randomly  accessed  and  individual  loading  from  primary 
to  secondary  storage  is  not  needed.  Thus,  the  user's  software  becomes 
simpler  by  virtue  of  this  proposed  memory  hierarchy  organization.  The 
bipolar  memory  is  used  here  because  the  bipolar  memory  is  the  memory 
device  with  the  fast  fetch-time  in  present  technology  [111].  The  dis- 
advantages of  using  bipolar  memory  are  the  higher  cost  and  power  needed 
than  the  core  memory. 

5-2.2  Control  Flow  of  the  Proposed  Computer  Archi tecture 

The  control  flow  for  the  proposed  computer  architecture,  CAIP,  is 
shown  in  Figure  5.15.  The  Task  Management  Processor  (THP)  allocates  jobs 
to  the  parallel  processor  (PP)  or  the  sequential  arithmetic  processor 
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Figure  Memory  hierarchy  of  designed  computer  architecture. 
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CONTROL  FLOW  OF  DESIGNED  COMPUTER  ARCHITECTURE 
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Figure  5.15.  Control  flow  of  designed  computer  architecture 
for  Image  processing,  array  processing. 
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(SAP)  by  means  of  the  tag  examination  routine.  The  user's  program  pro- 
vides a flag  to  the  parallel  processor  which  indicates  whether  the  task 
is  local  or  global.  The  locality  tester  examines  the  flag  and  initiates 
the  SIMD  or  MIMD  mode.  The  S1MD  mode  is  appropriate  for  local  tasks.  In 
this  mode  a single  instruction  is  executed  simultaneously  on  the  image 
points.  The  local  task  means  that  the  instruction  is  executed  on 


individual  data  within  an  image  window,  such  as  calculating  the  histogram 
of  an  Image  window.  For  global  task,  the  MIMD  mode  Is  employed.  The 
global  task  means  that  part  of  the  task  is  performing  one  kind  of  opera- 
tion and  other  part  of  the  task  is  performing  another  kind  of  operation. 
For  example,  in  the  task  of  evaluating  textured  and  nontextured  areas, 
some  processors  perform  second  order  statistical  texture  analysis  or» 
certain  window  and  some  processors  perform  first  order  mean  vector 
analysis  on  the  corresponding  windows.  The  global  task  conprises  of 


two  different  natures  of  subtasks.  The  outputs  from  the  Sequential 
Arithmetic  Processor  (SAP)  communicate  with  the  parallel  processor. 
Therefore,  the  SAP  may  support  the  PP  and  vice  versa.  Parallelism  of 
task  is  exploited  by  the  parallel  processor  (PP)  to  obtain  high  speed 
performance.  In  the  meantime,  the  operations  of  the  Sequential  Arith- 
metic Processor  (SAP)  are  pipelined  to  the  parallel  processor  under 
program  control  in  certain  tasks  which  can  be  decomposed  into  pipelined 
processing.  Therefore,  the  control  flow  of  this  architecture  exhibits 
both  parallelism  and  pipelining.  This  arrangement  has  not  been  In- 
corporated into  any  of  the  existing  systems  discussed  in  section  5.1. 
Since  two  types  of  multiprocessing,  parallel  processing  and  pipeline 


processing,  are  exploited  simultaneously,  this  contributes  to  high 
speed  performance  in  the  proposed  computer  architecture. 
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5.3  ANALYSIS 

5.3.1  Performance  and  Cost-Effectiveness  Tradeoffs 

The  Parallel  Processor  (PP)  of  the  proposed  computer  architecture 
consists  of  an  array  of  microprocessors  which  are  the  processing  elements. 

In  determinir.  the  optimal  number,  II,  of  microprocessors  for  the  parallel 

Processor  (PP),  a systematic  procedure  is  needed.  Such  a procedure  follows: 
in  designing  the  parallel  processor,  as  the  number  of  processing  elements 
(microprocessors)  increases,  the  number  of  data  points  processed  by  each 
processor  decreases  and  processing  speed  increases,  but  the  scheduling 
overhead  also  increases.  Therefore,  the  processing  speed  inprovenent 
readies  a saturation  point  at  a certain  number  of  proc'  35 ing  elements. 

So  it  seems  that  the  number  of  processing  elements  corresponding  to  the 
saturation  point  would  be  a good  choice.  However,  the  answer  ts  not 
that  simple,  as  cost-effectiveness  is  an  important  factor  in  the 
feasibility  of  a computer  architecture.  Thus,  the  costs,  such  as  hard- 
ware and  software  costs  of  the  parallel  p-ocessor  (PP) , need  to  be 
considered.  Hardware  cost  usually  involves  the  hardware  purchased  and 
the  cost  of  physical  construction  of  the  system.  Software  cost  reiers  to 
development  of  the  operating  svstems  and  software  supports  The  best 
choice  of  optimal  number  of  processing  elements  is  obtained  by  evaluating 
the  performance  inprovenent  and  cost  increment  on  different  image  pro- 
cessing tasks.  The  optimal  number  directly  depends  on  the  specifications 
which  are  given  by  the  user.  Let  us  use  the  following  hypothetical 
example  for  illustration,  if  the  processors*  cost  increases  with  the  Increase 
in  the  number  of  processing  elements  (microprocessors),  and  the  software 
cost  Increases  more  rapidly  than  the  processors*  cost  as  the  number  of  pro- 
cessing elements  increases,  the  performance  curve  becomes  saturated  at 
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80  processors  for  task  A,  and  70  for  task  B shown  in  Figure  5.16.  H 
; should  be  in  between  70  and  80,  The  .processors'  cost  increases  mor«  or  less 

linearly  with  the  number  of  processors  (if  fewer  than  100  processors  are 
bought).  But,  as  stated  above,  the  software  cost  increases  more  rapidly 
than  the  processors'  cost.  Jf  the  software  cost  increases  sharply  at  60 

processors,  as  in  Figure  5.1 7»  the  optimal  number  of  processors,  con- 
sidering cost  and  performance  trade-off,  is  betv/een  60  and  80,  Depending 
on  the  specifications  of  the  user,  if  the  concern  is  more  for  performance 
than  cost,  then  a number  near  80  is  chosen.  If  the  user  is  more  concerned 
about  cost,  then  a number  near  60  should  be  ciiosen. 

5.3.2  Implementation  of  Statistical  Methods 

As  discussed  in  Chapter  2 during  recent  years,  a number  of  image 
processing  algorithms  have  been  developed  (2,83].  In  this  section, 

( some  image  processing  techniques  are  discussed  In  terms  of  the  proposed 

computer  architecture,  CAiP,  to  exemplify  the  operations  of  the  special 
computer  for  image  processing. 

Statistical  texture  analysis  has  been  an  important  topic  in  the 
field  of  image  processing  [9,72,73].  The  texture  analysis  technique  in 
Chapter  b consists  of  histogram  equalization,  and  texture  feature 
measurement,  in  applying  the  proposed  computer  architecture,  CAIP,  to 
such  a texture  analysis  technique,  the  task  of  texture  analysis  is 
assigned  a tag  P (which  stands  for  parallel  processor  task)  by  the  user's 
program.  (Tag  S stands  for  sequential  arithmetic  processor  task).  The 
subtasks  of  texture  analysis,  such  as  histogram  equalization  and  texture 
feature  measruenent,  are  designated  by  the  flags  S!  (which  denotes  the 
SIMD  mode  for  the  task),  and  Ml  (which  denotes  the  MIMD  mode  for  the 
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task).  Assuming  the  size  of  the  image  is  MxM.  The  texture  analysis 
task  is  allocated  to  the  parallel  processor  (PP)  of  the  CAIP  by  the  tag 
examination  routine  of  the  Task  Management  Processor  (TMP).  The  Control 
Unit  of  the  Parallel  Processor  (CUPP)  finds  the  flag,  SI,  for  histogram 
equalization  and  loads  the  operating  system  of  the  SIMD  mode.  Each 
processor  of  the  N microprocessors  then  calculates  the  histogram  of  an 

d—  x picture  window.  The  outputs  of  the  ti  processors  are  then  put 

ft  »¥ 

together  to  obtain  the  equalization  result  of  the  final  histogram.  The 
CUPP  keeps  the  parallel  processor  (PP)  in  the  SIMD  mode  after  examining 
the  flag  SI  of  the  texture  feature  measurement  task.  For  example.  If  the 
88x88  picture  discussed  in  Chapter  li  is  to  be  processed,  each  processor 
will  process  the  co-occurrence  matrix  of  the  window  of  11x11  pixels. 

The  variability  texture  feature  measurement  is  calculated  by  each  pro- 
cessor as  the  texture  value  for  the  center  cell  (M)  of  that  window. 

The  mapping  of  the  array  of  processors  to  the  image  points  is  shifted 
four  pixels  and  repeats  the  texture  feature  measurement  task.  When 
this  shift  reaches  the  right  edge  of  image,  the  mapping  Is  shifted  four 
pixels  downward  and  the  process  is  repeated  from  the  left-most  column 
of  the  image.  This  process  continues  until  the  texture  values  of  all 
the  picture  points  are  obtained. 

5.3.3  Syntactic  Methods  and  Parallel  Processing 

As  has  been  pointed  out  previously,  syntactic  methods  for  image 
processing  have  increased  in  importance  for  certain  applications.  Pre- 
vious special  computers  such  as  the  PPM  and  the  PICAP  are  unable  to 
process  syntactic  methods  by  parallel  processing  [99].  However,  the 
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proposed  CA1P  furnishes  the  capability  of  parallel  processing  of  syntac- 
tic algorithm.  In  using  the  parallel  processor  for  syntactic  methods, 
parallel  parsing  schemes  are  most  desirable  as  they  can  utilize  the 
capability  of  the  parallel  processor.  Unfortunately,  the  research  in 
parallel  parsing  schemes  Is  very  limited.  In  this  section,  we  introduce 
the  parallel  parsing  of  tree  languages  and  explore  the  parallel  parsing 
of  the  parallel  context-free  languages  [103]. 

1 . Tree  Languages  and  Parallel  Processing 

The  Parallel  Processor  (PP)  of  the  proposed  computer  architecture 
for  image  processing  can  he  applied  to  tree  grammar  parsing.  The 
parallel  parsing  procedure  of  a tree  grammar  is  described  below.  The 
task  of  tree  grammar  parsing  is  designated  by  a flag  P and  tag  SI  by  the 
user's  program  for  the  effective  utilization  of  the  facilities.  Through 
the  control  unit  of  the  computer  architecture,  the  parallel  processor  is 
put  into  the  5IMD  mode  for  the  task  of  tree  grammar  parsing.  If  a pro- 
duction rule  of  the  tree  grammar  which  is  applied  to  parse  the  language 
has  k branches,  then  each  of  these  k branches  has  a nonterminal.  Each 
processor  or  the  Parallel  Processor  (PP)  is  assigned  by  the  user  to 
parse  one  nonterminal.  This  procedure  is  applied  to  consecutive  parsing 
of  the  language  until  a final  parsing  result  is  achieved.  If  the 
parsing  is  successful,  then  the  language  (pattern)  is  accepted.  If  the 
parse  fails,  then  the  language  (pattern)  is  rejected. 

For  example,  the  tree  grammar  Is  = (V,  r,  P,  S),  where  V = {S,  a, 

b,  $,  A,  B},  VT  =*  (*>a,  ib , $},  r (a)  = (2,  1,  0),  r(b)  = {2,  l,  0},  r (S)  =■ 

2 and  P:  [?.] 

I 

{ 

i 
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rule  1 : rule  2:  rule  3: 

S ->-  $ A a B b 

/ \ / \ / \ 
A B A B A B 

ru*e  ,i:  rule  5:  rule  6: 

A a B ->  b A a 

! 1 

b a 

This  grammar  generates  such  patterns  as 


In  order  to  perform  parallel  parsing  of  the  tree  language,  processors 
need  to  be  assigned.  First,  the  depth  "d"  of  the  tree  is  defined  as 
the  number  of  the  levels  of  the  tree.  The  depth  of  the  tree  In  (1)  is 
2 (d=2)  and  the  depth  of  the  tree  in  (2)  is  h (d=4).  The  maximum  number 
of  branches  for  all  the  tree  grammar  rules  is  easily  obtained  by  check- 
ing the  values  of  r in  the  grammar  Gt  «=  (V,  r,  P,  S)  and  this  number  is 
called  n.  The  relationship  between  n and  r is  that  m Is  the  maximum  of 
the  values  of  the  r’s.  The  number  of  needed  processors  in  the  parallel 
processor  (PP)  is  (d)m.  This  procedure  is  performed  for  the  worst  case 
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protection  concept  which  allows  the  maximum  number  of  branching  In  each 
level  of  the  tree  parsing.  If  the  number  (d)m  Is  greater  than  the  number 
of  processors  of  the  parallel  processor  (PP),  there  are  two  solutions: 
one  Is  the  static  priority  procedure,  and  the  other  is  the  dynamic 
priority  procedure.  The  static  priority  procedure  has  a fixed  priority 
rule  for  assigning  the  available  processors  to  the  proper  subtasks.  The 
fixed  priority  rule  for  the  parallel  tree  parsing  scheme  is  to  assign 
the  k left-most  branches  the  equal  priority  in  each  parsing  stage.  The 
number,  k,  is  the  number  of  available  processors  of  the  parallel 

processor  (PP)  for  each  parsing  stage.  At  parsing  stage  one,  the  k left- 
most branches  are  parsed  first,  based  on  the  highest  priority  rule.  At 
parsing  stage  two,  the  k left-most  branches  (nonterminals)  of  stage  two 
then  have  the  highest  priority  to  be  parsed.  Thus,  the  k available 
processors  are  assigned  to  parse  these  nonterminals. 

In  the  dynamic  priority  procedure,  a dynamic  priority  rule  has  to 
be  established  at  each  stage  of  parsing  in  order  to  determine  which  sub- 
tasks have  the  highest  priority.  For  example,  the  k left-most  priority 
could  be  assigned  first,  then  the  k right-most  priority  assigned  next 
at  the  request  of  the  user.  However,  in  parsing  the  tree  languages,  all 
the  individual  branches  (nonterminals)  have  to  be  parsed  to  get  the  nodes 
(terminals),  therefore,  the  static  priority  procedure  is  better.  The 
dynamic  priority  procedure  would  only  he  used  in  special  cases,  such  as 
the  case  in  which  only  a partial  parsing  result  is  of  Interest. 

Using  the  example  given  above  to  illustrate  the  proposed  parallel 
parsing  scheme  for  tree  languages  and  to  compare  the  parsing  result  with 
conventional  parsing  scheme  for  tree  languages,  if  the  depth  of  the  tree 
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to  be  parsed  is,  at  most,  four  - the  maximum  number  of  processors  needed 
is  easily  calculated  to  be  (1»)  = 16.  In  the  parallel  parsing  of  the 

input  tree  ci, 

/ 

$ 

/ \ 

a b 

/ \ / \ 
a b a b 

! / \ I 

a a b a 

i ! 

b a 

The  task  is  allocated  to  the  parallel  processor  (PP)  by  the  P flag  found 

by  the  Task  Management  Processor  (TMP)  and  the  parallel  processor  (PP) 
is  placed  In  the  SIMD  mode  by  CUPP.  The  procedure  is  graphically  illu- 
strated in  Figure  5.18.  At  the  parsing  of  depth  one,  one  processor 
parses  the  language  by  the  rule  1.  At  the  parsing  of  depth  two,  two 
processors  are  assigned.  The  number  of  processors  needed  for  the  parsing 
of  depth  k is  automatically  determined  by  the  number  of  branches  obtained 
form  the  parsing  of  depth  k-1.  The  number  of  branches  obtained  from 
depth  one  in  our  example  is  two.  Thus,  two  processors  are  needed,  one 
for  the  parsing  of  the  branch  starting  from  nonterminal  A and  the  other 

for  the  parsing  of  the  branch  starting  from  nonterminal  B.  Grammar 

rules  2 and  3 are  simultaneously  applied  by  the  two  processors  to  parse 
the  two  branches  of  the  tree  a starting  from  A and  B respectively.  At 
the  parsing  of  depth  three,  four  processors  are  needed.  Tv/o  processors 
simultaneously  parse  the  branches  of  A and  B which  are  the  result  of 
parsing  rule  2 '<n  the  depth  tv/o.  The  parsing  rules  for  these  two  pro- 
cessors are  rules  and  3,  respectively,  to  nonterminal  A and  B.  The 
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other  two  processors  simultaneously  apply  rules  6 and  5 to  parse  the 
branches  of  A and  B,  which  are  the  result  of  parsing  by  rule  3 In  the 

depth  two.  At  the  parsing  of  depth  four,  since  there  are  only  two  non- 

s 

terminals  A and  B left  from  the  parsing  of  depth  three,  two  processors 
are  assinged  to  simultaneously  parse  the  branches  A and  B by  rules  [i  and 
5 respectively.  Thus,  the  parallel  parsing  is  completed  and  the  tree  Is 
accepted.  The  parsing  of  the  same  tree  language  by  the  conventional 
sequential  parsing  scheme  is  shown  in  Figure  5*19.  At  each  parsing 
stage,  only  one  nonterminal  can  be  parsed  by  the  parser.  At  parsing 
stage  one,  grammar  rule  1 is  applied.  Rule  2 Is  applied  at  parsing 
stage  two.  Then  the  rules  *»,3, 4,5, 3, 6,  and  5 are  applied  to  parsing 

stages  3, A, 5, 6, 7, 3,  and  9 respectively.  Nine  parsing  stages  are  needed 
for  the  parsing  of  the  same  tree  as  shown  in  Figure  5.19.  It  can  be  seen 
in  Figure  5.18  that  only  four  parsing  stages  were  needed  for  parallel 
tree  parsing  scheme.  Thus,  in  this  example  there  is  a saving  of  over 
50?  in  parsing  stages,  and,  therefore,  a corresponding  saving  in  time  by 
utilizing  this  parallel  parsing  scheme  on  the  proposed  computer. 

2.  Parallel  Context-Free  Language  and  Parallel  Processing 

The  parallel  context-free  language  was  defined  by  Siromoney  and 
Krithlvason  in  [103].  The  definition  of  a parallel  context-free  language 
is  a language  generated  by  a context-free  grammar  in  which  the  manner 
of  applying  the  grammar  rules  Is  restricted  as  follows:  If  a nonterminal 
occurs  more  than  once  in  a sentential  form,  then  every  occurrence  of 
the  nonterminal  is  replaced  at  the  same  time  by  the  same  rule. 

The  parallel  processor  (PP)  of  the  proposed  computer  architecture 
for  image  processing  (CAIP)  performs  the  parsing  of  a parallel  context- 
free  language  in  the  SIHD  mode.  From  the  definition  of  a parallel 
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Figure  5.18.  Parallel  tree  parsing  procedure 
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Figure  5.19.  Conventional  tree  parsing  procedure. 
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context-free  language,  the  number  of  needed  processors  of  the  parallel 
processor  Is  equal  to  the  maximum  number  of  occurrences  of  a nonterminal 
in  all  sentential  forms,  A processor  is  assigned  under  programmer  con- 
trol to  each  nonterminal  when  it  occurs  simultaneously  with  the  same 
nonterminal  in  the  derivation.  These  processors  perform  the  parsing  of 
same  grammar  rules  on  these  nonterminals. 

For  example,  the  task  of  parsing  a parallel  context-free  language  is 

assigned  the  flag  P and  tag  SI  which  initiate  the  SIMD  mode  for  the  task. 

2n 

The  parallel  context-free  language  is  L (G)  s {a  Jn  > 0}  [1033-  The 
parallel  context-free  grammar  is  G = (V,  !,  P,  $)  v/here  V - {S,  a}, 
i = {a},  and  P = {S  -*■  SS,  S ■*  a).  If  the  languages  to  be  parsed  are  aa 

and  aaaa.  The  maximum  number  of  occurrences  of  nonterminals  in  all 
sentential  forms  is  four  which  comes  from  aaaa  (nonterminal  ssss) . Thus, 
the  number  of  needed  processors  is  four.  At  the  first  stage  of  parsing 
of  the  language  aaaa,  one  processor  is  assigned  to  parse  the  language  and 
the  grammatical  rule  is  S SS.  At  the  second  stage  of  parsing,  two 
processors  apply  the  same  rule  5 -*•  SS  on  the  two  nonterminals  "S"  and 
the  parsing  result  is  SSSS.  At  the  third  stage  of  parsing,  four  pro- 
cessors apply  the  same  rule  S ->  a on  all  the  four  nonterminals  **S**. 

Hence,  the  parsing  result  Is  aaaa.  This  language  cannot  be  passed 
sequentially  as  the  language  is  defined  to  be  parsed  only  parallel!’/. 

The  sentence  aaaa  of  this  example  is  parsed  by  the  parallel  context- 
free  grammar  G.  Thus,  the  sentence  is  accepted  as  a member  of  L(G). 

In  the  preprocessing  part  of  the  tree  grammar  approach  in  Chapter 
4,  the  task  of  horizontal  and  vertical  processing  is  assigned  a flag  MI 
which  denotes  the  MIHD  mode  for  the  task.  The  horizontal  and  vertical 
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processing  task  is  thus  allocated  to  the  parallel  processor  (PP)  of  the 
CAIP  by  the  tag  examination  routine  of  the  Task  Management  Processor 
(TMP) . The  control  unit  of  the  parallel  processor  (PP)  loads  the 
operating  system  of  the  MIMO  mode  after  finding  the  HI  flag  of  the  task. 
The  parallel  processor  has  N processors.  Half  of  the  set  of  ti  processors 
Is  designated  to  execute  the  horizontal  processing  and  the  other  half  of 
that  set  of  processors  executes  the  vertical  processing.  The  different 
instructions  are  executed  on  multiple  picture  data  at  the  same  time  by 
the  parallel  processor  (PP) . In  this  example,  the  computer  architecture 
CAIP  utilizes  the  multiple  instruction  stream  multiple  data  stream  (MIMD) 
mode  to  achieve  high  system  throughput  for  this  task. 

5.b  SUMMARY  AMD  REMARKS 

A computer  architecture  for  image  processing  (CAIf)  has  been  pro- 
posed. This  computer  architecture  is  designed  by  the  distributed 
computing  approach.  This  computer  is  comprised  of  a parallel  processor 
(PP)  and  a sequential  arithmetic  processor  (SAP).  The  flexibility  and 
high  performance  of  this  computer  architecture  are  contributed  to  by  two 
major  features  which  are  the  reconfiguration  capability,  described  in 
section  5-2.1,  and  the  method  of  computer  exploitation  of  task  parallel- 
ism, stated  in  section  5.2.2.  This  computer  architecture  for  image 
processing  is  proposed  to  use  microprocessors  as  the  processing  elements. 
The  advantage  of  using  a microprocessor  array  is  that  the  cost  of  micro- 
processors is  much  lower  than  that  of  conventional  processors.  The  dis- 
advantage is  that  the  processing  power  of  microprocessors  is  less  than 
that  of  conventional  processors,  especially  In  addressing  capability. 

For  example,  the  most  popular  microprocessors  INTEL  8080  and  MOTOROLA 
6800  do  not  have  associate  addressing  or  microprogramming  abilities. 
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For  special  inage  processing  computers,  some  processing  powers  of  con- 
ventional processors,  such  as  associative  addressing,  are  not  essential 
[102].  The  approach  of  designing  a general  purpose  computer  by  micro- 
processors has  been  controversial.  But,  in  designing  a special  purpose 
computer  for  image  process  fng,  microprocessors  have  their  advantages. 
Furthermore,  the  advance  in  semiconductor  technology  Is  toward  the 
development  of  microprocessors  with  higher  processing  power.  The  recent 
developments  in  the  microprocessors  servies  ItlTEL  3000,  MOTOROLA  M29O0, 
and  Texas  Instruments  7^81  have  provided  microprogramming  capability 
to  microprocessors. 

With  the  fast  growth  of  image  processing  and  its  applications,  the 
need  for  a special  image  processing  machine  such  as  the  proposed  computer, 
CAIP,  should  certainly  be  appreciated. 
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CHAPTER  6 

C OHC LUS IONS  AND  SUGGESTIONS  FOR  FURTHER  WORK 
6.1  SUMMARY  AND  CONCLUSIONS 

In  this  study,  we  have  presented  the  finite-state  string  grairmar 
approach  for  image  recognition,  a tree  grammar  approach  for  image  seg- 
mentation, and  a special  computer  architecture  for  image  processing. 

The  system  for  image  recognition  v/as  presented  in  Chapter  3.  This 
system  consists  of  preprocessor,  syntactic  analyzer,  and  postprocessor 
[118],  Two  methods  were  presented;  namely,  syntax-directed  [7]  and 
syntax-controlled  methods.  The  finite-state  string  grammar  was  applied 
to  the  recognition  of  highways  and  rivers  from  LAHDSAT  images.  For  the 
syntax-controlled  method,  the  finite-state  string  grammar  was  automati- 
cally inferred  by  the  k-tail  finite-state  grammatical  inference  proce- 
dure. From  the  experiments,  the  finite-state  string  grammar  which  is 
inferred  by  the  k=2  case  in  the  grammatical  inference  procedure  was 
found  to  be  the  most  suitable  of  those  investigated  for  highway  and 
river  recognition.  Some  further  applications  of  the  syntactic  method 
were  bridge  and  conmercial/industrial  area  recognition.  The  method  ex- 
tracted the  structural  and  contextual  information  from  the  images  to 
recognize  the  objects  of  interest.  The  locations  and  lengths  of  bridges, 
as  well  as  the  centers  and  sizes  of  commercial/industrial  areas  were 
extracted  by  the  appropriate  algorithms  described  in  Chapter  3. 
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In  Chapter  b,  a syntactic  method  for  Image  segmentation  was  presented. 
This  method  is  a tree  grammar  approach  which  utilizes  a tree  automaton  to 
extract  the  boundaries  of  the  homogeneous  region  segments  of  an  image. 

The  homogeneity  of  the  region  segment  was  obtained  through  the  texture 
feature  measurements  of  the  image.  The  experiments  were  conducted  on 
different  Images  obtained  from  satellite,  and  Infrared  sensors.  The 
results  of  syntactic  image  segmentation  compare  favorably  with  those  of 
statistical  classification  techniques. 

In  Chapter  5,  a computer  architecture  for  image  processing  was 
proposed.  This  computer  architecture  was  designed  by  the  distributed 
computing  approach.  This  computer  consists  of  a parallel  processor  and 
a sequential  arithmetic  processor.  Two  major  features  which  are  new  to 
the  field  of  special  purpose  computer  architectures  for  image  processing 
contributed  to  the  flexibility  and  high  performance  of  this  architecture. 
These  features  are  the  reconf igurable  capability,  described  in  section 
5.2.1,  and  the  method  of  computer  exploitation  of  task  parallelism, 
given  in  section  5.2.2. 

In  conclusion,  the  syntax-controlled  method  for  image  recognition 
was  found  to  be  more  powerful  than  the  syntax-directed  method,  commonly 
known  as  template  matching.  Firstly,  the  computer  processing  of  the 
recognition  process  for  the  syntax-controlled  method  is  faster  than  that 
of  the  syntax-directed  method.  Secondly,  the  recognition  by  the  syntax- 
controlled  method  is  based  on  an  automaton  (or  parser)  which  is  much 
more  powerful  and  flexible  than  recognition  by  matching  the  templates  in 
the  syntax-directed  method.  Also  the  recognition  power  of  the  syntax- 
directed  method  is  limited  by  *-he  number  of  templates.  The  advantage 
of  the  syntax-directed  method  is  its  fast  software  development  time 
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which  means  the  time  of  developing  and  Implementing  such  a method  is 
fast  and  less  complicated  than  the  syntax-controlled  method.  However, 
once  the  software  of  the  syntax-controlled  method  is  developed,  the 
syntax-controlled  method  saves  program  execution  time  in  performing  the 
task  every  time. 

The  selection  of  an  appropriate  grammatical  approach  for  recognition 
usually  depends  on  the  problem  requirements  [2].  The  string  grammar  is 
most  suitable  in  describing  a string-like  pattern,  such  as,  highways  and 
rivers.  Thus,  the  string  grammar  approach,  which  uses  a finite-state 
automaton  as  syntactic  analyzer,  is  applied  for  the  image  recognition  in 
Chapter  3- 

For  the  problems  In  Chapter  h,  the  patterns  of  interest  are  the 
boundaries  of  the  image  segments.  These  boundary  patterns  are  high 
dimensional.  The  tree  grammar  is  more  convenient  in  describing  high 
dimensional  objects  than  string  grammar,  therefore,  a tree  grammar  was  used. 
The  tree  grammar  offers  a natural  high  dimensional  generalization  of  strings 
and  the  tree  automaton  has  a high  analytical  capability  in  recognizing 
patterns.  Thus,  the  tree  grammar  approach,  which  utilizes  a tree  auto- 
maton as  syntactic  analyzer,  is  applied  for  image  segmentation  in  Chapter 

4. 

We  believe  that  the  syntactic  algorithms  for  image  recognition  and 
segmentation  developed  in  Chapter  3 and  A provide  a better  way  to  under- 
stand image  structure  and  to  extract  image  information  than  these  have 
previously  been  done.  These  syntactic  algorithms  can  be  used  for 
military  reconnaissance,  industrial  automation,  and  medical  diagnosis. 

In  addition,  the  fast  growth  of  image  processing  and  its  Impact 
on  industrial,  biomedical,  and  military  applications,  has  created  a need 
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for  a special  image  processing  computer  such  as  that  proposed  In  Chapter 
5,  namely,  the  computer,  CAIP. 

6.2  SUGGEST  I QMS  FOR  FURTHER  WORK 
Syntactic  algorithms  and  special  computer  architectures  for  image 
processing  are  very  important  areas  of  research.  There  are  still  many 
open  problems  that  are  worth  investigating. 

The  finite-state  string  grammar  and  the  tree  grammar  approaches 
need  to  be  further  developed  for  more  and  varied  applications.  Other 
grammatical  approaches,  such  as  a context-free  grammar  [2]  and  a parallel 
context-free  grammar  [103]  need  investigation  for  their  applications  to 
image  processing.  Parallel  parsing  schemes  for  finite-state,  context- 
free,  and  context-sensitive  grammars  need  to  be  studied.  And  the 
special  computer  described  in  Chapter  5 needs  to  be  physically  construct- 
ed according  to  the  proposed  computer  architecture  in  Chapter  5,  and  put 
into  operation. 
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APPENDICES 


APPEND  I X A 

SAMPLE  PATTERNS  FOR  INFERENCE  OF  THE 
TREE  GRAMMAR  FOR  IMAGE  SEGMENTATION 


APPENDIX  B 

THE  FLOW  CHART  OF  HIGHWAY  RECOGNITION  VIA 
THE  SYNTAX-DIRECTED  METHOD 


i 

l 


APPEND  IK  C 

THE  FLOW  CHART  OF  RIVER  RECOGNITION  VIA 
THE  SYNTAX-DIRECTED  METHOD 


Real  world  digitized 
satellite  images 

(Infrared  bands,  channel  3 and  k) 


Use  the  found  THRESHOLD  R in  the  inference 
process  for  "water-like  area"  to  threshold 
the  images  to  binary  image. 

greater  than  or  equal  to  the  THRESHOLD  -*■  0 
less  than  the  THRESHOLD  -*■  1 


Transformation 

process 


Line  smoothing  process 


String  grammar  analyzer  to  accept  the  patterns 
which  are  generated  by  the  string  grammar, and 
reject  other  patterns 


String  grammar 
analysis 


River  recognition  result  via  the 
syntax-directed  method. 


APPENDIX  D 

THE  FLOW  CHART  OF  BRIDGE  DETECTION  VIA  THE  SYNTAX- 
DIRECTED  METHOD  WITH  SEMANTIC  PROCESS 


1 
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Real  world  satellite  Images 


The  flow  chart  of  highway 
recognition  by  the 
syntax-directed  method 


The  flow  chart  of  river 
recognition  by  the 
syntax-directed  method 


_J 


Implementation  of  semantic  rules 


Calculate  the  number  of  horizontal  rows  having 
bridge  points,  the  number  to  be  ,,an 


Calculate  the  number  of  vertical  columns  havinc 
bridge  points,  the  number  to  be  "b" 


The  length  of  bridge 
is  "c" 

c * /(a  * 79)2  + (b  * 56)2 


The  length  ofj 
bridge  is  c 
= b * 56 


Locate  the  coordinate 
of  bridge 


Bridge  recognition  result 
via  the 

syntax-directed  method  with 
semantic  process  t-* 


APPENDIX  E 


TRAINING  SAMPLES  FOR  HIGHWAY  GRAMMAR 


237 


c-\MP!.r  P-i  T T~_ 

1 1 1 1 1 1 1 l 

0 0 0 0 0 0 0 0 

0 0 0 0 C 0 0 ) 

ty  r,  n r ' 

• • • ^ ■ ' 

0 0 P 0 f,  '!  .1 

0 o A .w  . 

nooo  ;•  •- 

,o  o o o o ' ) 


o o n o o o o i 

0 0 n 0 0 j 1 0 

0 0 p 0 0 1 0 I) 

0 0 1 0 o ' 

n o r>  i -j  * ) 

0 0 I 0 0 C *.•  .) 

0 1 o 0 0 <1  0 0 

10  0 0 0 0 0 c 


1 0 C 0 0 i;  ' "• 

1 0 0 n 0 0 ! P 

1 0 0 0 C 0 0 •' 

1 0 0 0 0 l ••  •' 

1 0 0 0 0 ■)  s 1 

10  0 0 0 O’  0 > 

1 (.  0 0 0 •»  . 

i o A "o  ; , • 


11111110 

0 f:  0 0 0 0 0 L 

0 0 o 0 0 0-0  0 
0 0 0 0 0 0 0 0 

0 C 0 Z C 0 0 0 

n o o r.  o -o  i;  -o 

0 0 0 0 o 0 0 0 

n r,  t 0 0 C 0 


i -y  • r>  •:  ) 

o i <-•  > o 

0 0 1 0 j : .1 

A A ' :•..>•=  ‘ 
; 1 ■ . • o 

a ' \ '>1  1 

•'  * '•  '''  vi  i .. 

0 r 0 ; • 


0 0 0 0 0 '■  0 1 

0 •?  " 0 •?  I 0 

1 I 1 i 1 l 0 : 


ft  <■  O 


■ ■ <_ 


0 0 r ■’  0 0 0 c 

0 c ? 0 0 i 0 0 

0 0 c 0 0 M 0 J 

0 0 P <i  C {: 


0 


233 


1 

n 

r 

0 

0 

0 

0 

l 

i 

i 

i 

1 

0 

0 

•) 

A 

‘J 

1 

0 

"> 

0 

;) 

u 

.) 

0 

0 

0 

0 

0 

l 

0 

>3 

') 

r 

1 

i 

i 

1 

l 

1 

1 

o 

n 

0 

0 

0 

0 

l 

A 

o 

0 

0 

0 

0 

o 

a 

0 

0 

r. 

0 

0 

0 

0 

0 

1 

0 

0 

A 

c 

r. 

*.y 

A 

r, 

0 

C 

n 

0 

A 

.} 

V 

- 

• J 

•s 

•) 

0 

3 

5 

fi 

A 

2 

A 

r. 

t 

% 

„ 

' 

- 

W 

V.* 

A 

0 

o 

0 

0 

A 

A 

n 

,■> 

? 

3 

0 

e 

(; 

n 

0 

0 

l> 

0 

0 

n 

r 

•> 

3 

r 

t 

>•']']!!!] 
’ :■  < ) ? > '•  . 
n-  0 3 o 0 '•  .•  • 

n ^ 0 r-  c '» 

? 0 0 r*  0 C j •*. 

0 0 r,  0 0 0-'  '• 

C -■>  0 0 0 ; 

0 3 r,  ) o a 

1 t'  o o j i 

n I 0 0 ) ; 

o o i A ^ ; , 

0 0 *»  l i! 

n r ,A  : I i ; i 

(I  ' 0 0 \ : i 


>)  0 is  C C 0 “ I 

1 <•  0 0 n 0 2 >) 

0 C •!  0 0 I u r, 

0 0 3 a i 1 -1  .| 

1 1 1 l <•  0 ■)  ! 

o o 3 a o o .»  j 

0 0 A ,A  £ ,,  p,  p; 

o o o o 3 : .)  „ 

n o l i i i } 
o i •)  o a o v 

0 t ; 3 0 0 

1 ? 3 1 3 L - 1 

r 0 ft  0 ? 1 

r.  n o 0 •*• 


r-  ' -i  ; , • : 1 


0 n " : 0 


' r*  p ^ r 

“ V/  ’ 


0 «.■  n 0 5 0 


239 


BEST  AVMIAB 


}\l 

i>LL 


rAf \\j 


1 1 1 0 0 0 0 0 

0 0 0 1 0 0 0 0 

0 0 0 0 1 c 0 0 

o o o r.  o i o o 

0 0 ft  o 0 0 1 .* 


0 C 0 0 0 l 1 1 

o o o o i o o 

0 fi  0 1 {<  .)  0 0 

0 0 1 0 ? j 0 G 

o 1 r 10  0'  '> 


r, 


ft 


,1 


1 ">  0 0 0 ^ ft 


0 0 '.1  0 0 


0 


0 o ft  0 0 .)  J 


f)  ft  ft  r,  n f\ 


i 


i 


i 


1 0 0 0 j }■ 

0 1 0 0 0 0 ',.- 
■0010  C 0 ' • 

o o o l n o • 

0 0 0 0 1 0'* 

0 r-0  0 0 1 1 *■ 

r>  o o o •**  :•  i ; 

1 r-  o o o j ■ 


i o o o : ->  i 

0 10  0 0 0 0 
0 0 1 1 1 l ! • 

0 0 0 0 0 ! .t  i 

0 0 0 0 0 0 :}  0 

0 0 C 0 0 0 0 0 

o r.  o o o o n o 

o c-  o o o a o •> 


0 o o 0 0 0 ..  1 

r.  n o o n ) \ ,■ 

ft  0 0 !*)  ft  1 ' >*. 

0 0 0 0 1 '• 


ft  ft  ft  3 0 0 o 1 

0 0 o »:  o o 1 <» 

0 111110) 

1 o « *»  o , o 


f 0 *'  1 o 


0 o ft  ? 0 


\ 


r, 


1 


ft 


o o n r*  o 


o ft 


0 J .0  0 0 0 - 0 

n r>  o o 0 ft  10 


0 


I'  Vf(lt  i!"'1  ffW*  **  *v 


1 

p 

:: 

P 

J 

f: 

.ft 

J 

J 

0 

0 

o 

1 

i 

1 

1 

1 

1 

i 

L 

0 

P. 

•/ 

0 

0 

•1 

0 

0 

} 

p 

V/ 

0 

o 

0 

n 

r 

V 

i 

p 

/ 

p 

0 

p 

1 

1 

0 

0 

0 

0 

Q 

0 

p 

0 

0 

1 

7 

0 

u 

0 

r? 

0 

0 

ft 

o 

./ 

n 

p 

f 

i' 

r 

\ 

j 

1 

, 1 

n 

p 

a 

p 

'•J 

•J 

P 

. ■ 

■\ 

ft 

"i 

A 

r * 

■J 

£ 

n 

p 

t\ 

p 

c 

p 

ft 

i 

0 

r. 

■i 

P 

P 

0 

0 

i 

0 

o 

Ct 

n 

/ 

j 

0 

n 

') 

0 

P 

fft 

J 

0 

0 

0 

Q 

p 

r\ 

p 

; 

} 

P 

.1 

1 

/• 

P 

1 

; 

ft) 

a 

I 

1 

r. 

0 

0 

a 

0 

ft 

n 

A 

*> 

0 

0 

- 

i 

0 

i 

0 

o 

r 

v> 

0 

■j 

0 

' 

p 

r 

.*\ 

- 

- 

.j 

V 

V 

“ 

r 

0 

1 

" 

n 

0 

O’ 

A 

r, 

[) 

p 

r 

» 

P 

. 

- 

“ 

1 

f! 

ft 

0 

1 

Q 

#ft 

0 

•j 

p 

1 

A 

1 

p 

i 

1 

• i 

« 1 

0 

0 

0 

ft 

l 

p 

c 

••) 

ft 

* 

r 

A 

0 

P 

, ', 

0 

vJ 

0 

0 

0 

0 

i 

r» 

ft 

■) 

P 

p 

i ; 

P 

- 

p 

a 

' 

** 

i 

0 

•/ 

ft 

-» 

,n 

p 

/ 

1 

0 

i' 

0 

, 

- 

P 

a 

** 

, 

* 

j 

M 

V 

0 

0 

0 

n 

l 

t) 

0 

0 

1 

1 

1 

l 

a 

1 

l ft 

**•  p 
> 

n 

0 

n 

ft 

n 

c 

j 

ft 

P 

<y  ] 

: i 

0 

0 

o 

7 

0 

♦> 

1 

ft 

r. 

f; 

• 

l i 

n 

o 

0 

7 

] 

-ft 

, . 

(ft 

l\ 

p 

0 

•*j 

n 

(> 

t 

P 

l 

»! 

•> 

ft 

A 

r, 

/* 

ft 

* 

% % 

\ 

0 

>.) 

n 

«ft 

■j 

1 

U 

'* 

•i 

A 

. i 

•• 

\ 

■i  .. 

0 

0 

■3 

0 

7 

.) 

J 

,i 

P 

P 

ft 

* 

) 

•J  .. 

n 

ft 

p 

\ 

1 

A 

v i 

t 

" 

•i 

0 

p 

0 

n 

V 

1 

: 

ft 

APPENDIX  F 

SAMPLE  PATTERNS  FOR  INFERENCE  OF  TREE 
GRAMMAR  FOR  MILITARY  VEHICLE 





APPENDIX  G 


THE  FLOW  CHART  OF  TREE  GRAMMAR  INFERENCE  PROCEDURE 


Input  sample 
pattern  1 


CALL  ENCODE 

Encode  the  pattern  Into  primitives 
a,  b,  c,  d,  e,  f,  g,  and  h,  and  use 
pointers  to  form  the  linked  tree 
data  structure 


Check  the  values  which  arc  pointed 
the  pointer  of  the  tree.  Group  the 
encoded  tree  into  subtrees  with  re- 
petitive substructures  (some  values). 


Take  the  root  of  each  subtree  and  the 
succeeding  value  linked  by  the  pointer. 
Delete  rest  of  the  tree. 


Start  from  the  root  of  the  input  tree, 
replace  the  values  which  are  linked  by 
the  values  corresponding  to  the  non- 
terminals. Then  the  grammar  rule  is 
inferred  and  stored  in  computer  as  a 
linear  array. 


A1 1 subtrees  are 
checked? 


i=n  1 

n:  no,  of  sample, 


Output 


r 


APPENDIX  H 


THE  FLOW  CHART  OF  TREE  GRAMMAR  ANALYSIS 





/Boundary  Primitive 
Extraction  Result 
nxn  picture 


i = 

=1 

3 

C 

• 

J= 

=1 

CALL  ENCODE 

Encode  the  pattern  into  linked  tree 
structures  TREE(I1,JJ)  with  the 
primitives  a,  b,  c,  d,  e,  f,  g,  and  h. 


Construct  tree  automaton  TA(L,M) 


L= 

i 

: 

-^fA(L,2)  = input  symbol^' 
(TREE (TREE ( 1 1 , J J+k) , 1)) 
"'"M:  = pointer  number^''"'" 


L=L+1 


Check  next  state 
TAV=TA(L,3) 


^fnd  transition  ruTfc''*-^ 
for  next  level 
— IA(LL  .Jj^TAiW- 

Yes 


LL-LL+1 


.z'' 
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THE  EXAMPLE  OF  TREE  AUTOMATON  CONSTRUCTION 





T 
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Example  Grammar  from  section  4.1.1. 

Gt  - (V,  r,  P,  S) 

^ V = (Aj,  A2,  S,  $,  c,  d,  e,  f) 

V.j.  = {|,  -*c,  m d,  +e,  ^f} 

r(4)  « £1}  r(c)  ={0,1,2}  r(d)  = {2} 

r(e)  = {0,1}  r(f)  = {0,1} 


P = 


S ->  4 

1 

A 


1 


A,  ■*  c 
1 / \ 

A1  A2 


A2 


/\ 

A,  A, 


A, 


c 

I 

A, 


e 

I 

A, 


A. 


The  detailed  steps  of  tree  automaton  construction  are  as  follows:  the 
tree  automaton  is  Mt,  «t  = (Q,  fc,  fg,  fd>  f^,  F) , where  Q.  = {q£,  qe»  qr 


q?,  qs>  and  F = {qc,qe}. 


Grammar  Rule  of  Grammar 

S -v  4 

I 

A1 

S 4 

i 


Transition  Rule  of  Automaton 
f^qI)  “ qs 


fi(q2>  " ■'s 
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Ai  * 

c 

/\ 

A1  A2 

fc(q,,<I2)  - q, 

s 

A2* 

d 

/ \ 

A1  A2 

fdlql’q2)  = q2 

Al  * 

c 

1 

W = ql 

A1 

A„  -*■ 

/. 

e 

1 

f0(q2>  ' q2 

A2 

a2- 

e 

fe(qe>  " q2 

A1  " 

c 

fetqe!  ' ql 

Thus 

the  tree 

automaton  is 

constructed 

as 

• "t  = <«•  fc-  fe> 

where 

0 « {qc 

• V ql  > "2' 

qs}’  F s {qc 

.,q  } and  f: 

• C 

(1) 

VV  = qs 

(5) 

fc(q|>  “ ql 

(2) 

f}(q2>  = qs 

(6) 

fe(q2>  “ q2 

(3) 

fc(Vq2>  " 

ql 

(7) 

W " q2 

(4) 

td(ql’q2)  ’ 

q2 

(8) 

W " q, 
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